"PFF employs over 600 full or part-time analysts, but less than 10% of analysts are trained to the level that they can grade plays. Only the top two to three percent of analysts are on the team of “senior analysts” in charge of finalizing each grade after review."
1) You are making assumptions about the roles of part-time employees and applying them to the entire company.
No, I'm saying that the grades, which are done by poorly trained individuals paid very little to do it, are nothing more than a completely unknown mix of player reputations and players stats. Saying that more-highly trained individuals check the grades is silly. If they had time to check them carefully, they could do them themseleves.
And the result is there to see. It's really easy to find player grades for games that are complete nonsense.
2) PFF grades over a trailing period are good predictors of their future PFF grades—on par with EPA, QBR, etc.
There's definitely useful information buried in PFF grades, and you could probably extract some of it if you had no better source. But since we know the grades are just some mix of player reputations, player stats, and actual player performance in the games being graded, but in completely unknown proportions (but probably roughly in the order stated here from highest to lowest contribution, but there's no way to know), if you want to understand how a player actually performed in a game, you're better off looking at stats and something like a "sentiment analysis" of what people are saying about the player on Xitter or certain web sites.
Player reputation is a good predictor of future player reputation. Some stats are good predictors of future values of themselves. PFF grades are some unknown mix of player reputation, player stats, and of course the individual preferences of the graders (some might hope that last part washes out on average, but it can be a big mess when looking at the grades for individual games). So of course PFF grades are a decent predictor of their own future values, because they're made of some unknown mix of things that are good predictors of their own values.
What I'm saying is that PFF grades are not leading indicators of anything. There's no trend like "wow, PFF has been grading these guys highly for a long time while they produced little in stats and got little attention, and now they're blowing up. PFF really identified these good players that stats and reputation didn't!" There's also no opposite trend ("uh-oh... I know these guys are stars, have really good stats for their respective positions, and are projected to keep being really good, but PFF is saying they're bad, so look out... their performance is about to start falling off cliffs." When a PFF grade is different from other ways of measuring performance, that's not a leading indicator that the other measures are about to change too.
PFF grades give you no useful information that stats and reputation don't, and it's harder to separate those things than if you just used them directly instead of paying somebody else to mix them up with evaluation of what the player did in the game performed by poorly trained people getting fifty bucks per complete game charted, and then checked by people with more training, but without time to re-grade every player on every play of every game.
x.com
3) The real issue is that most people don't understand how to read PFF grades. For example, 50 is the normalized expected average—such that a player average of 50 over the season would be neither a positive nor negative to their team. Thus, the further away from 50 in the positive direction, the more impactful the player.
A quality starter typically falls around 65 (This is roughly Jarran Reed's career average), so Murphy being a 68 is actually a good sign. Again, though, the sample size from one game is mostly useless.
So the thing is, the right way to read that last part is not "Jarran Reed gets an average grade around 65, so he's clearly of NFL-starter quality." It's more like "Jarran Reed is a regular NFL starter, so the PFF graders ended up giving him average grades around 65." PFF grades are more trailing indicators of stats and rep than leading indicators of anything except themselves. There's good information mixed in unknown proportions into PFF grades, but there are better sources of the same information without the clutter of bad reputation-based and stat-based grades that pretend they're additional information beyond stats and reputations like professional scouting reports would be.
.
