Why you shouldn't put too much stock into PFF Grades

[AgentDib]

PFF grades are just an entertainment product but they also invoke a cognitive illusion where we erroneously feel like the more information we consume, the more signal we receive. Taleb discusses this at length in Antifragility and it is summarized as the Noise Bottleneck here.

I believe that PFF ratings are less leading indicators of player quality and much more trailing indicators of players' statistical performance and reputations.

That's a better way of stating my earlier point. In the context of this thread, Murphy was clearly dinged because he didn't finish the sack (production). He was also being double teamed on that play and still managed to penetrate up field and be disruptive (actual performance). I would bet a lot that Seahawk coaches gave him at least a plus on that rep based on his actual performance.

 

[morgulon1]

Their actually stats are WAY more interesting than their grades. Grades are subjective. Stats are not.

I think stats are completely subjective for some positions.

CB Richard Sherman's interceptions went down dramatically after a couple of seasons. Why ? Because he sucked? It was because he terrified the opposition and for the most part ignored his side.

A DT generally don't generate sacks , yet they hold the center of the field, collapse the pocket and push the QB out towards the DE.

I think maybe there's a happy middle ground.

 

[GemCity]

Agree with OP. Subject matter expertise combined with statistics can tell a story but, I take PFFs stuff with a grain of salt.

Example…Knight was the second highest, PFF graded Hawk in our first preseason game. I thought he did well but, there were a few plays that had he attacked his assigned gap, he would’ve had some TFLs. Defensive schemes that were spot-on…but not executed other full extent.

I imagine MM schooling him a bit about his hesitancy on attacking his gap.

Still impressed with him but his PFF stats have him looking much better than reality.

 

[knownone]

I'm not sure if you were referring to me or to the people dismissing PFF because it's "subjective." My point was that PFF grades are worth little, but it's not because of "subjectivity." As I mentioned, there are subjective elements and human choices that come into even the "hardest" statistical analysis.

PFF has a lot more noise than professional scouting reports. There's still useful information that could be extracted from PFF ratings if that were all you had, but in the context of having a lot of recorded objective information about the game (yes, with subjective choices behind what information is chosen to be recorded), and given how PFF collects its data (see below), I believe that PFF ratings are less leading indicators of player quality and much more trailing indicators of players' statistical performance and reputations.

How PFF collects its player-rating data
PFF pays people $50 per complete game to look at every play and give players ratings based on what each geting-fifty-bucks-for-the-work evaluator, without knowing the play calls, thinks each player was supposed to do on each play, and to get and keep the "job," each evaluator is required to spend, in addition to NFL-watching time, six hours or more watching college football on Saturdays. These aren't trained professionals. These are people who respond to a web cattle call to make fifty bucks for five or six hours of rating player performance (or more quickly but even less reliably), and they're willing to waste their Saturdays watching college football in order to get paid $50 per NFL or college game charted. Yikes.

Remeber "GIGO"? "Garbage in, garbage out." In the context of how PFF gets its data, "GIGO" a pretty good description of what to expect from PFF player ratings and rankings.

1) Why are you comparing PFF to professional scouting? It is a consumer-facing product and none of us work for a professional football team.

2) You are looking at a description for part-time, probational employees. All of their grades are reviewed by full-time, trained employees.

3) Similarly, your argument is that low pay = garbage data. That's just an assumption. In reality, a strict criterion with reviews and consistent application is what matters. Know your ruler, and you'll understand what it's measuring. How much a part-time employee earns is irrelevant.

The consistency of PFF's methodology is more important than the potential "noise" in their data. In other words, as long as PFF consistently applies its grading system and guidelines across all evaluations, the presence of noise in individual grades is insignificant for the average fan.

 

[BigMeach]

PFF grades were up within 15 minutes of games ending last season. If that doesn't tell you all you need to know about PFF, I don't know what does?

 

[Lagartixa]

1) Why are you comparing PFF to professional scouting? It is a consumer-facing product and none of us work for a professional football team.

Because PFF advertises its grades as if they were something with even the reliability of professional scouting reports (and if you follow the NFL draft carefully, you know that there's a lot of noise in professional scouting reports).

I've also seen defenders of PFF say stuff along the lines of "well, NFL teams subscribe to PFF," as if those grades are what teams want.

2) You are looking at a description for part-time, probational employees. All of their grades are reviewed by full-time, trained employees.

If those full-time, trained employees had time to review the grades carefully, they could just do the grading themselves. But PFF continues to put out cattle calls for people desperate enough to want 50 bucks for what's either several hours of work or just half-assed bullshit to get the 50 bucks.

3) Similarly, your argument is that low pay = garbage data. That's just an assumption. In reality, a strict criterion with reviews and consistent application is what matters. Know your ruler, and you'll understand what it's measuring. How much a part-time employee earns is irrelevant.

That's the thing. Nobody has any idea what PFF's "ruler" (player grades) is actually measuring. It's some bizarre mix of what actually happened in plays in the game being graded (tho' the grader has to decide to what extent a player did what he was supposed to do on each play without knowing the play call and how player responsibilities were assigned on the play), player reputations, and player stats, plus other unknown factors. I've never seen PFF grades shown to be a leading indicator of anything except themselves. Since some of the components we know are involved to unknown degrees in PFF grades (reputation, stats) are decent predictors of themselves, it's not at all surprising that PFF grades are reasonable predictors of future PFF grades. But I am unaware of even a single case in which PFF grades disagreed with stats and reputation, and then were shown to be more right about the player in question than his stats and rep had been.

The consistency of PFF's methodology is more important than the potential "noise" in their data. In other words, as long as PFF consistently applies its grading system and guidelines across all evaluations, the presence of noise in individual grades is insignificant for the average fan.

The thing is that PFF grades are presented as bringing new information, independent of the players' reputations and stats, to player evaluation. The reality is that they're an unknown mix of what graders actually saw and player reputations and stats, and it really looks like PFF grades are much more redundant trailing indicators of what stats and reputation can tell you about a player than leading indicators of anything except themselves. Even when they disagree, like when Ryan Neal was the highest-graded safety in the NFL in 2022, it's the PFF grades that come back to agreeing with the stats rather than the stats developing to agree more with the PFF grade. The fact that Neal got $1.2M for one year from the Bucs after being rated as the best safety in the league by PFF suggests strongly that teams know PFF grades are not valuable for player evaluation. And lookie what happened: Neal's 2023 PFF grade came back down in line with his stats and reputation.