Why you shouldn't put too much stock into PFF Grades

[Maelstrom787]

I've found that PFF's grading system, in general, usually meets the eye test. For example, last season, they had both Adams and Diggs ranked in the bottom tier of safeties and guess what? They both really sucked. And guess what? They aren't Seahawks anymore.

Diggs, according to the film guys who actually knew the scheme, really wasn't bad though. He was still adept in coverage. Tackling was never his strong suit, but he was also being put in a position to tackle far more often than he should have as the free safety.

Most of the deep Seahawks film nerds seem to be in agreement that Diggs was fine (not perfect - but still a good player) and covering for deficiencies in Wagner's coverage game coupled with deficiencies in the general run defense.

Diggs isn't getting a terrible deal, either. $3M guaranteed with incentives to push it to 5 is pretty good for a 31 year old safety on a prove-it deal. Also, the Titans DC happens to be Mike's DB coach in Baltimore last year.

 

[chris98251]

He didn't suck at all.

Oh he sucked, only average, Guess instead of taking on triple teams he should have just taken on the whole O line and made tackles and a sack all in the same play, oh and picked up the officials flag for the O line holding him blocking all 5.

 

[BigMeach]

Byron Murphy was noticeably absolutely pummeling the Chargers backup guards, regularly pushing them deep into the pocket and causing Easton Stick to panic. This was consistent and it absolutely flew off the screen. The Chargers ended up double and triple teaming him, which he *still* beat a few times.

55 pass rush grade and 26 tackling grade.

With how quickly they put up the grades it is clear it's 100% stat based and nothing to do with visuals whatsoever. We who watched the game saw how dominating he was.

 

[Lagartixa]

This is the 1 main reasons I don't do the stat thing. They are completely misleading and do not tell the reality of what actually happens on the football field.

Tell us you don't understand statistics without explicitly saying "I don't understand statistics."

PFF is subjective nonsense with a lot of money spent on advertising and promotion to give people the impression it's useful and capitalize on the fact, known by even most casual fans, that teams are now using "analytics" to make better team-construction decisions.
Correctly used statistics in team sports like NFL football is the the application of probability theory (in one of what basically ends up coming down to two basic ways) to get at individual-player contributions to team success based on the objective data recorded about what happens on the field. The fact that mediots misuse statistics doesn't invalidate the use of statistics for the evaluation of players and plays in specific game situations. One thing that sets better NFL front offices and coaching staffs from weaker ones is that the better ones make better use of good statistical analysis.

 

[Lagartixa]

I've found that PFF's grading system, in general, usually meets the eye test. For example, last season, they had both Adams and Diggs ranked in the bottom tier of safeties and guess what? They both really sucked. And guess what? They aren't Seahawks anymore.

Even a blind squirrel can find a nut.
Even a blind pig can find a truffle.
A broken analog watch is exactly correct twice a day (but good luck figuring out when those two moments are).

PFF ratings basically come down to "the individuals paid very little to evaluate players in this game like some of those players and dislike others, and they think some of those players are good and others not, based on their reputations." I'm not saying PFF grades aren't correlated with actual on-the-field performance. What I'm saying is that they're more of a trailing indicator of reputation than a leading indicator of actual performance.

 

[SoulfishHawk]

Byron Murphy was noticeably absolutely pummeling the Chargers backup guards, regularly pushing them deep into the pocket and causing Easton Stick to panic. This was consistent and it absolutely flew off the screen. The Chargers ended up double and triple teaming him, which he *still* beat a few times.

55 pass rush grade and 26 tackling grade.

great post man, and spot on

He was pushing fools around. He has some real good leaders mentoring him too. They were clearly having fun with him after missing that sack. He took it in stride, and by all accounts, he's a pretty humble kid.

 

[RiverDog]

Diggs, according to the film guys who actually knew the scheme, really wasn't bad though. He was still adept in coverage. Tackling was never his strong suit, but he was also being put in a position to tackle far more often than he should have as the free safety.

Most of the deep Seahawks film nerds seem to be in agreement that Diggs was fine (not perfect - but still a good player) and covering for deficiencies in Wagner's coverage game coupled with deficiencies in the general run defense.

Diggs isn't getting a terrible deal, either. $3M guaranteed with incentives to push it to 5 is pretty good for a 31 year old safety on a prove-it deal. Also, the Titans DC happens to be Mike's DB coach in Baltimore last year.

That's true, and the same can be said about Woolen, that poor play from our front seven exposed his weakness more than it otherwise would have, and yes, one does have to keep in mind those kinds of unquantifiable subjective factors when looking at something like a rating from PFF.

But that's true of almost every statistic or graphic. You have to read between the lines. It doesn't mean that they're worthless or irrelevant as some are claiming.

As far as Diggs goes, it wasn't just his poor tackling that was an issue. He was a slacker, didn't even want to tackle, shied away from contact. I don't want that kind of player on my team, and apparently, Macdonald felt the same way.

 

[SoulfishHawk]

I liked Diggs, until I didn't. He looked out of position and/or like he didn't care. I'm not saying he didn't care, clearly, he does. But damn, he was bad last year.

 

[uncle fester]

It amazes me that anyone looks at PFF.

They made up their own grading system. Chris Collinsworth - the majority stakeholder in PFF - sells it to the viewers as important during the game. How that isn’t a conflict of interest is beyond me.

PFF is about as much use as asking the mascot how the game went.

Back on topic; that Chargers center wont be looking forward to seeing the tape today…

 

[Fresno Hawk]

Tell us you don't understand statistics without explicitly saying "I don't understand statistics."

PFF is subjective nonsense with a lot of money spent on advertising and promotion to give people the impression it's useful and capitalize on the fact, known by even most casual fans, that teams are now using "analytics" to make better team-construction decisions.
Correctly used statistics in team sports like NFL football is the the application of probability theory (in one of what basically ends up coming down to two basic ways) to get at individual-player contributions to team success based on the objective data recorded about what happens on the field. The fact that mediots misuse statistics doesn't invalidate the use of statistics for the evaluation of players and plays in specific game situations. One thing that sets better NFL front offices and coaching staffs from weaker ones is that the better ones make better use of good statistical analysis.

Your point being.

 

[sutz]

Kind of like political polling. It's an indicator, but not a tell all/be all. They try to be "objective" but that's hard to do. Unlike baseball, football has small sample sizes and a lot more variables that can have unpredictable effects on the data.

I don't mind people quoting PFF, but it's not gospel, just another set of data to use in evaluating players.

 

[RiverDog]

Their actually stats are WAY more interesting than their grades. Grades are subjective. Stats are not.

That works well for a quarterback or receiver, but there's not too many stats for an offensive guard.

 

[RiverDog]

Kind of like political polling. It's an indicator, but not a tell all/be all. They try to be "objective" but that's hard to do. Unlike baseball, football has small sample sizes and a lot more variables that can have unpredictable effects on the data.

I don't mind people quoting PFF, but it's not gospel, just another set of data to use in evaluating players.

Precisely!

In baseball, every pitcher has an ERA, every fielder a fielding percentage, and every hitter a batting average. But how do you quantify a defensive tackle or a center? Football is a very difficult sport to quantify player performance, some positions more than others.

PFF makes an attempt to rate performances that are very subjective. They look at and evaluate every player on every play of every game, so they do have a very large sample size, at least for those players who get a lot of PT. A lot of football people who know a lot more about the game than we do reference it.

PFF is not gospel, but it's not worthless, either.

 

[Nosferatu]

That works well for a quarterback or receiver, but there's not too many stats for an offensive guard.

Very true. But the grades are still subjective. You can definitely look at things like average gain per play while in, average gain per play to your side, sacks, negative yards, etc. Just watching it and trying to assign a grade is so subjective.

 

[RiverDog]

Very true. But the grades are still subjective. You can definitely look at things like average gain per play while in, average gain per play to your side, sacks, negative yards, etc. Just watching it and trying to assign a grade is so subjective.

Ave. yards/play your side doesn't work so well if you pancake your man and the guy next to you whiffs or if the RB doesn't hit the hole. I haven't seen a stat yet that accurately represents the performance of an offensive lineman.

Sure, grading is subjective. But if you get two people who know a decent amount about the game, you and me for example, to give the same player on a given play a score of 1-10, you're not going to grade it a 0 while I grade it a 10. We're going to be within 2-3 points of each other. It's not rocket science.

The effect of biases and poor evaluators can be mitigated by rotating evaluators so that each player gets evaluated by a different evaluator every game or even every quarter. I know that's how I'd run it if I had anything to do with it.

 

[Lagartixa]

Very true. But the grades are still subjective. You can definitely look at things like average gain per play while in, average gain per play to your side, sacks, negative yards, etc. Just watching it and trying to assign a grade is so subjective.

There is a lot of subjective stuff that still goes into player evaluation. Professional scouts are human beings with biases like all of us have. Choices about how to measure the value of specific in-game events (which stats) are subjective. If they're based on some specific objective criterion, the specific choice of what criterion to use is subjective.

The "Bayesian" approach to statistics allows for subjective expert knowledge to be included as part of the initial guess at how likely a quantity of interest is to take on different specific values, known as the "prior." One of the most influential Bayesian statisticians of all time, Bruno de Finetti, considered probabilities to be completely subjective, and many Bayesians treat probability theory that way today. But don't think that using frequentist statistics (the other major "school" of statistic - the one with all the stuff from Fisher, Neyman, Pearson... that gang) will get you away from subjectivity. In all of parametric frequentist statistics, there's the subjective choice of a likelihood function, and that's just as subjective as any prior (or likelihood!) in Bayesian statistics.

The choices of the tools to use are subjective. You can't get away from subjectivity.

The thing is that there's value in subjective data. Scouting reports are a great example. There's a lot of noise, but there's some really interesting "signal" there too.

Modern tools give us a bunch of ways of including subjective information in models. Think of "Twitter sentiment trading," for example. A program can analyze what is being said on Xitter (Twitter when the term was invented) about a specific stock or currency or something else and generate additional signals to a trading application. Similarly, models of NFL rookie performance can take into account the content of (text) scouting reports on players.

Pulling together scouting reports and analyzing them as predictors of NFL player performance, modern tools also allow us to detect and correct for individual scouts' preferences and biases.

The thing about PFF ratings is that there's more noise than in, for example, professional scouting reports. I'm sure there's useful information in there, but I wouldn't even try to figure out how much of it is based on actual observation and how much of it is based on players' stats and reputations. Like I said elsewhere, I wouldn't be at all surprised to learn that PFF ratings are more lagging indicators of reputation than leading indicators of performance and value, and I'd expect a wide range of statistical models to give better estimates of the value of previous performance and infinitely better predictions of what to expect next from a given player (PFF grades are descriptive and not in any way predictive).

 

[knownone]

Subjectivity is a buzzword people throw around to dismiss things lately. The core argument is that having humans involved in a process in some way invalidates the end product. While many of the points are technically accurate, it's an incredibly tiresome argument when almost everything is subjective.

The question, then, is not whether a tool is subjective, but rather, how subjective? PFF uses a standardized criterion with consistent application across every play with multiple checks and balances. It is not someone watching a play and randomly pulling a grade out of his/her butt. Thus, it is relatively objective, ceteris paribus. As long as you interpret their grades within that context, knowing it's not an objective measurement for ranking players but instead an evaluation of past performance using a specific criterion by a specific evaluator. It has value.

 

[RiverDog]

Subjectivity is a buzzword people throw around to dismiss things lately. The core argument is that having humans involved in a process in some way invalidates the end product. While many of the points are technically accurate, it's an incredibly tiresome argument when almost everything is subjective.

The question, then, is not whether a tool is subjective, but rather, how subjective? PFF uses a standardized criterion with consistent application across every play with multiple checks and balances. It is not someone watching a play and randomly pulling a grade out of his/her butt. Thus, it is relatively objective, ceteris paribus. As long as you interpret their grades within that context, knowing it's not an objective measurement for ranking players but instead an evaluation of past performance using a specific criterion by a specific evaluator. It has value.

Excellent post.

Even so-called objective statistics, such as a quarterback's completion percentage, has a certain degree of subjectivity to it. There's a difference in the completion percentage of a quarterback playing in a game where he's trailing by 4 TD's in the 4th quarter vs. in a tie game. The defense plays looser, they only rush 3, give him the underneath stuff, blah blah blah. Or how about quarterback sacks or tackles? Who makes that decision and what kind of criteria do they use?

J/B data is collected by subjective means doesn't mean that it isn't credible or valid.

 

[Lagartixa]

Subjectivity is a buzzword people throw around to dismiss things lately. The core argument is that having humans involved in a process in some way invalidates the end product. While many of the points are technically accurate, it's an incredibly tiresome argument when almost everything is subjective.

The question, then, is not whether a tool is subjective, but rather, how subjective? PFF uses a standardized criterion with consistent application across every play with multiple checks and balances. It is not someone watching a play and randomly pulling a grade out of his/her butt. Thus, it is relatively objective, ceteris paribus. As long as you interpret their grades within that context, knowing it's not an objective measurement for ranking players but instead an evaluation of past performance using a specific criterion by a specific evaluator. It has value.

I'm not sure if you were referring to me or to the people dismissing PFF because it's "subjective." My point was that PFF grades are worth little, but it's not because of "subjectivity." As I mentioned, there are subjective elements and human choices that come into even the "hardest" statistical analysis.

PFF has a lot more noise than professional scouting reports. There's still useful information that could be extracted from PFF ratings if that were all you had, but in the context of having a lot of recorded objective information about the game (yes, with subjective choices behind what information is chosen to be recorded), and given how PFF collects its data (see below), I believe that PFF ratings are less leading indicators of player quality and much more trailing indicators of players' statistical performance and reputations.

How PFF collects its player-rating data
PFF pays people $50 per complete game to look at every play and give players ratings based on what each geting-fifty-bucks-for-the-work evaluator, without knowing the play calls, thinks each player was supposed to do on each play, and to get and keep the "job," each evaluator is required to spend, in addition to NFL-watching time, six hours or more watching college football on Saturdays. These aren't trained professionals. These are people who respond to a web cattle call to make fifty bucks for five or six hours of rating player performance (or more quickly but even less reliably), and they're willing to waste their Saturdays watching college football in order to get paid $50 per NFL or college game charted. Yikes.

Remeber "GIGO"? "Garbage in, garbage out." In the context of how PFF gets its data, "GIGO" a pretty good description of what to expect from PFF player ratings and rankings.

 

[Lagartixa]

Excellent post.

Even so-called objective statistics, such as a quarterback's completion percentage, has a certain degree of subjectivity to it.

You're right about that.

There's a difference in the completion percentage of a quarterback playing in a game where he's trailing by 4 TD's in the 4th quarter vs. in a tie game. The defense plays looser, they only rush 3, give him the underneath stuff, blah blah blah.

That's just the fact that any given stat is more useful in some contexts than others. The subjectivity comes into the picture even earlier, in the decisions about what statistics to record in order to measure player performance. Even if you choose the stats to record and use in analysis based on some "objective" criterion like "use stats that, when included in models, help minimize the mean square error of our predictions," there's still a subjective choice in the use of mean square error instead of mean absolute error or some other criterion. Any kind of statistical modeling involves human choices about how to build the model.

The problem with PFF is the ancient computer-science maxim "GIGO" (garbage in, garbage out). PFF's data are collected by paying people $50 per game to give a rating to each player on each play, based on what the making-fifty-bucks-for-it person thinks each player is supposed to do on each play without knowing the play call. The quality of the data is severely limited by being produced by people willing to take $50 for several hours of work, and who are desperate enough to get those fifty bucks that they're willing to commit to at least six hours every Saturday watching college-football games and giving player ratings for each player in each play in those games. To be fair, I should say that I think evaluators get an additional $50 per college game charted too, but how reliable do you expect data to be when they're generated by people who are desperate enough for the money that they're willing to chart three or four games per week for $150 or $200? They're either spending about a half-time job's worth of hours for $150 or they're half-assing the work to get it done faster. This is why I would be really surprised if PFF ratings are a leading indicator of, well, anything, and I would expect them to be trailing indicators of players' statistical performance and reputation.

There's definitely useful information buried in PFF ratings, but I would be really surprised if you couldn't get the same information weeks earlier by looking at players' stats and reputations (run a "sentiment analysis" on what people are saying about the player on Xitter or in articles on the web, for example).