Greg passed along this comment regarding the Oilers’ ESTR’s:
“Without looking into it further, it really seems to benefit players who played a lower qualcomp on their team with very good linemates (Ryan, Oshie and Berglund come to mind there), but there are some very good players who look good by a lot of metrics (Datsyuk, Hossa, Havlat). Nilsson played with good players against the dregs and finished +2. Horcoff played with the best against the other teams best and finished +9. Yet it’s ranking Nilsson higher?”
A fine question, and a good case study to undertake. Concentrating on the Oilers’ portion of the question, let’s compare Edmonton forwards Shawn Horcoff and Robert Nilsson. At first glance, counting stats seem to be a hands down victory for Horcoff:
| Name | GP | G | A | P | +/- |
| Shawn Horcoff | 80 | 17 | 36 | 53 | +9 |
| Robert Nilsson | 64 | 9 | 20 | 29 | +2 |
But Horcoff played in 80 games versus Nilsson’s 64 games, and got 21:21 minutes versus 15:11 minutes (ATOI). Let’s look at what difference that makes to the rates if we give Nilsson equal games and ice time:
| Name | G | A | P | P/GP | +/- |
| Shawn Horcoff | 17 | 36 | 53 | +0.66 | +9 |
| Robert Nilsson* | 16 | 35 | 51 | +0.63 | +4 |
| *Normalized to Horcoff’s GP and ATOI | |||||
All of a sudden, the two players are as equal as you can get, on counting stats and points per game rates, right?
Yeah, except one thing. Horcoff still would have seen more PPTOI (300 to 277 minutes) than Nilsson even with the normalization (The base numbers were 300 minutes to 158 minutes PPTOI).
Oh, and another. At Even Strength, taking out Empty Net goals, both players were +6, and that’s without normalization.
And by our metrics, the average GF and GA for Nilsson were under slightly more difficult circumstances than for Horcoff.
So how do these numbers look now?
| Name | Pos | ESTOI | ESOR | ESDR | ESTR |
| Robert Nilsson | C | 810 | +0.33 | +0.15 | +0.48 |
| Shawn Horcoff | C | 1146 | +0.08 | +0.20 | +0.27 |
Hey Timo,
Thanks again for responding to this question. To me (I’m still getting my feet wet with advanced stats), it appears that both Horcoff and Nilsson had better linemates than the opposition that they faced at even strength (stats from behindthenet). As for Nilsson having more difficult circumstances than Horcoff for GF/GA, is that based on the Qualcomp/Qualteam numbers?
I’m not sure how you did your math to get those numbers, but let’s go through it the long way.
Total icetime (EV/PP/SH) & Total points (EV/PP/SH)
Horcoff:1145.5/263.4/299.9 & 32/21/0
Nilsson: 810.38/3.28/158.25 & 18/11/0
Now, let’s toss out the short-handed time; we do this because it isn’t a place where guys put up points (Kyle Brodziak led the Oilers with 2 SH points). Now, it’s probably indicative that Horcoff was on the PK and Nilsson wasn’t, but let’s leave that aside.
If we normalize Nilsson’s rates to Horcoff’s ice-time, we get the following comparison:
Horcoff: 32+21 = 53 total
Nilsson: 25+21 = 46 total
Your mistake, I think, was counting Horcoff’s SH ice-time as EV ice-time, when clearly it’s not comparable in terms of expected offense.
As for Nilsson playing under more difficult conditions than Horcoff, speaking as someone who watched the matchups all year that makes no sense. But let’s run the numbers:
Horcoff: .04QC – .18QT = -.14
Nilsson: -.03QC – .10QT = -.13
That looks pretty comparable until you realize where the faceoffs were being taken.
Horcoff: 328 def. vs. 239 off. = +89 defensive zone draws
Nilsson: 129 def. vs. 172 off. = -43 defensive zone draws
which leaves Horcoff with a net deficit of 132 draws.
Thanks, Greg and Jonathan. I wasn’t intending to do an exact calculation, but to use rough calculations with more common statistics to show that the ESTR numbers aren’t that crazy.
We don’t use Qualcomp and Qualteam. I did glance at those numbers over at http://www.behindthenet.ca when preparing my blog, as a sanity check.
Our numbers are close as well. Here’s what they say about the two forwards:
Even Strength GF – Horcoff GF in a 2.50 (per 60) scoring environment, Nilsson GF in a 2.46 scoring environment
Even Strength GA – Horcoff GA in a 2.46 (per 60) scoring environment, Nilsson GA in a 2.49 scoring environment
Both indicate that Horcoff had it slightly easier on both ends of the ice.
The faceoff stat is interesting -Clearly, you know your stuff- but my initial stab at faceoffs (see my ESPN Insider article and Puck Prospectus re-issue “This one goes to 11″ – archives) showed that even strength faceoffs brought about goals at a very low rate. I would assume that Horcoff had more offensive zone draws on the power play -much, much more likely to result in goals- so I’d venture to guess that the effect of faceoffs evens out more than it looks like on the surface. But I’m willing to be convinced otherwise…
Jonathan, thanks for coming over and sharing your thoughts. From my quick calculations, I didn’t really see a difference either with the Qualcomp/ Qualteam differential which is why I had asked how how Nilsson’s GF/GA were under more difficult circumstances.
Timo, I had recently put a link to this thread over at Lowetide’s blog. Lowetide and Jonathan Willis are the two most well-regarded Oilers bloggers–just to provide that introduction, so Jonathan does know the Oilers very well.
Timo:
Horcoff had more PP minutes, but I don’t see why he would have had a higher rate of offensive draws over those minutes than Nilsson.
How are your ESTR numbers calculated?
Timo: Nevermind on the earlier question.
After reading your methodology in the other post, I have to ask this question, though: do you make any effort to adjust for on-ice sv% and sh%, or no?
”
Even Strength GF – Horcoff GF in a 2.50 (per 60) scoring environment, Nilsson GF in a 2.46 scoring environment
Even Strength GA – Horcoff GA in a 2.46 (per 60) scoring environment, Nilsson GA in a 2.49 scoring environment
”
how did you determine these numbers?
are you saying that the average GF/60 for players playing against Horcoff is 2.46 and for Nilsson 2.49?
or are you saying that the oilers averaged 2.46 GA/60 while Horcoff was on the ice?
Great discussion. Thanks again for getting us started, Greg.
Everyone wants a peek at the engine, huh?
Jonathan – Goalies are factored in, as other skaters. I understand that may at first seem a bit controversial, but you end up with the net effect of various goalies (as well as the skaters on defense), without having to worry about the components of why. I think it’s an elegant solution.
Q – It’s the combined rates of teammates and opposition on the ice.
Everyone wants a peek at the engine, huh?
I assume that this is a reference to the fact that the methodology is black box. Yes, I suspect that everyone does want a peak at the engine. I look at everything to come out of PPro with a lot of skepticism (except for Gabe) because so much of it is black box. It’s not even worth discussing or reading the black box stuff, IMO.
Personally, if the methodologies aren’t published, I’m not going to be inclined to take the stuff seriously.
I have written a LOT about how it works, so that’s not a fair statement at all (There’s an intro article at Puck Prospectus as well if you’re interested). The basics are simple, the devil’s in the details.
@ Greg:
Horcoff had better teammates than competition?
He played Iginla, Sedin, Thornton, Zetterberg and Gaborik every night…. who in the blue hell were his teammates, and why aren’t they showing up in the box scores?!
Past that though, there’s a huge problem with these numbers… I’ll just copy and paste my response at LT’s blog…
Huge logistical fallacy still Noob:
He’s prorating Nilsson to Horcoff ice time.
While assuming Nilsson gets the soft minutes, and Horcoff gets his tough minutes.
… see the problem?
There are only so many times you can throw someone over the boards against the other teams 4th line.
It’s a huge handicap on Horcoff and a huge plus to a guy playing limited minutes (which explains why guys like Ryan and Nilsson are so highly ranked).
I came home after 12 beers and figured this out…
I’m assuming that most of you all commenting here watch every minute of Edmonton Oilers games – I do not. Therefore, I’ve got my ears open to hear what you perceive “outside the stat”: Who got the lucky bounces, who did the little things to help or hurt the team, who takes a night off sometimes, etc.
Similarly, some of you should take a look at what I’ve written and ask for clarifications before shooting from the hip with assumptions – some of which have been quite wrong (e.g. ESTR not accounting for strength of opposition).
Again, the point of the last blog was to do a “back of the envelope” calculation based on standard stats to see if we couldn’t at least convince a layman that it’s not unreasonable to compare Horcoff and Nilsson. One of the most basic mistakes many the average fan makes is leaning on counting stats, without seeing that players with less opportunities may be just as good.
The whole point of ESTR is to provide a combined stat, taking into account time on ice and quality of teammates and quality of opposition. As Greg quoted me over on his blog:
“Even Strength Total Rating (ESTR) is a more advanced version of Goal Difference per 60 minutes (GD/60), taking into account the quality of teammates and quality of opposition for all non-empty net Even Strength goals scored while a player is on the ice.”
It is pretty common for 1st liners to play 1st liners and 4th liners to play 4th liners. When that happens, even before running any calculations, you expect it to be a wash. When ESTR says that Nilsson had a slightly tougher scoring environment than Horcoff, that’s what it means, in layman’s terms. Yeah, Horcoff may play against Iginla and other 1st liners more often, but he does so with better linemates and likely the #1 defensive pair backing him up more often than Nilsson.
The majority of you will hopefully see ESTR’s value. Frankly, what you may be missing is not in its faults, but its limitations. By looking at ESTR alone, you don’t get the full picture of the player, as it says nothing about PP, SH or net penalty performance (Our stats for those are factors are getting their final touches).
@PDO–thank you for stopping by to share some thoughts from the Lowetide blog link for this topic. There’s no question that Oilers fans are both very passionate about the game and knowledgeable about their team. I wasn’t trying to stir a heated debate, but wanted to provide Tmo with some ‘outside the stat’ perspective on our hockey team (The Oilers).
In response to your question, my comment was in regard to the Qualcomp/Qualteam numbers at 5v5 play from behindthenet–Jonathan had posted them in his comment above. Shawn Horcoff at 5v5 play –(Qualcomp=0.04/Qualteam=0.18).
cheers,
Greg
@Timo – I realize this may have been mentioned elsewhere but I’m new to the conversation. If ESTR is accounting for quality of opposition, how is it doing so?
The reason I ask is that Qualcomp and Qualteam are extremely situational – If the Horcoff/Hemsky/Penner line breaks even against Iginla’s line, they all get a Qualteam of zero and Iginla gets a Qualcomp of zero.
If the same line goes out and runs up the score against Calgary’s dregs, Penner/Horc/Hemsky get awesome results for Qualteam, and Prust gets a high positive Qualcomp.
So if I’m Iginla and I’m surfing Behindthenet.ca the first thing I ask myself is why my Qualcomp is significantly lower than Prust’s, when both Iginla and Prust were facing the exact same lineup.
And the answer is that QualComp and Qualteam are relative to the player. They don’t measure how good the players are that you line up against, they attempt to measure how much better or worse the players are against you.
In that sense, it’s entirely possible for Nilsson to face “tougher” competition than Horcoff, because Nilsson isn’t as good of a hockey player. It’s easier for him to be overmatched. But if you’re using it to value a player, you’re using it wrong, unless “gets overmatched a lot” is a skill you value.
It’s more properly used to evaluate how a coach is matching his players – if a fourth liner has a break-even qualcomp it’s because he was playing fourth liners. If Zach Stortini has a Qualcomp of +0.50 it’s not because he’s a shut-down centre, it’s because Quinn couldn’t be bothered to protect him from the Sedins or Iginla.
Which is a very roundabout way of asking – does your measurement suffer from the same effect?
I should clarify the example used in my last post.
Imagine a league where the Oilers and Flames are the only two teams, and they play each other 82 games per year.
Each team has four lines of forwards that always play together as a group of three and only change on whistles. Every game has 60 minutes of 5v5 time.
Scenario 1: The coaches match lines. First line always plays first line, fourth line always plays fourth line. The teams are equally good, so every line breaks even.
Using the behindthenet.ca methodology, we must consider the difference between the +/- of the first line (zero, they broke even) and the +/- when that line isn’t on the ice (zero, the other lines also broke even).
Because the difference is zero for any player, Qualcomp and Qualteam will also be zero for every player (since you’re simply adding up and averaging a bunch of zeros).
So if Horcoff and Stortini both have a Qualcomp of zero, does that mean that Iginla and Prust are equally difficult to play against? Of course not.
Scenario 2: Coaches match second and fourth lines, but each first line matches against the other team’s 3rd line. Each first line outscores the third line by 2 goals per game.
Now the difference is massive – +2 in 15 minutes is +8/60. -2 in the remaining 45 minutes is -2.67/60. So each first liner is +10.67/60 and each third liner is -10.67/60.
So Glencross plays against the first line, and his Qualcomp is +10.67. Horc’s Qualteam is +10.67. Same deal if you flip it and talk about Iginla versus Pisani.
But the players haven’t changed – why does Iginla get a Qualcomp of 0 playing against Horcoff but Glencross gets a Qualcomp of +10.67? It’s not because Horcoff has magically gotten better. The stat tells you nothing about how good Horcoff is, or how good Glencross is.
All it tells you is that there is a chasm of skill between them. Do we applaud Horcoff and Iginla, and boo Pisani and Glencross? Or do we blame the coaches for matching this way?
And what if we sort by Qualcomp?
Pisani = +10.67
2 line = 0.0
4 line = 0.0
Horcoff = -10.67
Does that tell us that Zach Stortini, playing against Prust, faced better hockey players than Horcoff, playing against Glencross?
Of course not. All it tells us is that Stortini versus Prust isn’t nearly a mismatch like the other lines.
And that’s the long version of why measuring quality of opposition seems doomed to fail. I welcome your input as to how you’ve avoided this conundrum.
And I haven’t even addressed the biggest flaw with Qualteam, which is that in my above example, anybody playing with Pisani would have a massively negative Qualteam, because you have to average in his -10.67/60.
Which makes it much too easy to skim the stats and say “well this Stortini seems like a good teammate, but good god it must suck to have to drag this Pisani fellow around like a boat anchor”.
When we all know that everybody should want to play with Fernando. All the time.
Schitzo – Thanks for the detailed comments and questions.
(Been a busy “day off” – I took part in a fantasy hockey experts’ mock draft over at ESPN.com this afternoon. I’m sure that will be posted in the near future. Although not yet.)
In brief: We adjust ES GF/60 and GA/60 (without ENG) for each player by the (GF or GA) rates of all other players on the ice. It’s not relative to the player…except in the sense that the player himself is not included in the measure of goal difficulty. e.g. It wouldn’t be fair to Ovechkin to downgrade his GF/60 because Ovechkin was on the ice for the goals.
Schtizo,
Understand I am not a huge fan of QualComp or QualTeam, however if Iginla faces Pisani the whole year then his competetion level is significantly worse. QualComp doesn’t tell you how good ro bad a player is, but the competetion level they faced.
If the first line plays the first line, they had equal competetion against them, so why not a QualC of 0? Its not saying Iginla and Prust are equal talent, just their numbers were equally influenced by their competetion. Its not saying how good or bad Prust or Iginla is.