Looooong read on PFF

[AFJF]

Found this after discussing PFF in another thread and this is more recent than the article I found earlier this off-season.

For the TLDR crowd, 40% of NFL teams are using PFF data and the article mentions some specific coaches who use it.

Former HC/Coordinator Gunther Cunningham had a player complaining about his grade on PFF and thought it was unfair...he checked the grade that player was given by the team and PFF was "on the money".

As the previously posted article stated from Chris Collinsworth but repeated again here:

“What really impressed me is the fact that 13 NFL teams have contracted with Pro Football Focus for their data,” Collinsworth told The MMQB last August, after becoming a business partner. “I have been around the NFL for over 30 years, I know how hard it is to get behind the wall of those teams. And they’ve got 13 teams to trust their data. That’s huge.”

No idea what they're doing, but 13 NFL teams working with them?  Hmmmm.

In fairness, it also mentions the fact that some NFL coaches don't believe in it.

 

http://mmqb.si.com/2015/01/25/pro-football-focus-nfl-neil-hornsby-cris-collinsworth-analytics

 

CINCINNATI — In a mostly empty basement in suburban Ohio, save for two bookshelves filled with NFL record books dating back to 1942, an Englishman scribbles the names of pass rushers on a whiteboard. Cam Jordan. Jason Pierre-Paul. J.J. Watt. The guy holding the marker knows nearly everything about each snap these defensive ends played this season. Where they lined up. What the situation was. How well they did. Now he’s trying to answer the most incisive question: What does all this information mean? 

 

Neil Hornsby isn’t doing this solely as a labor of love, though that’s precisely how his business, Pro Football Focus, began nine years ago in Luton, England, some 30 miles north of London. Living stateside since October, he now counts 13 NFL teams—40% of the league—as clients, including seven teams from this season’s playoff field. He also oversees a staff of roughly 80 full- and part-time employees who watch countless hours of game footage from their home offices in California, Northern Ireland and seemingly everywhere in between.

 

There’s a good chance you’ve heard of Pro Football Focus. Last summer, former NFL wideout Cris Collinsworth made a significant investment in the company, and he trusts the analysis so much that he uses its scouting information when commentating on NBC’s Sunday Night Football. PFF’s statistics are widely cited in NFL media reports about players, teams and trends, and Hornsby has appeared in several stories on The MMQB. But what the public sees on Pro Football Focus’s website is just a tidal pool compared to the ocean of information that NFL teams are paying very good money (PFF won’t disclose how much) to access.

 

It’s more than a data dump. Pro Football Focus will meet virtually any of its clients’ requests, such as measuring hang time to two decimal points for punts and kickoffs, and tracking which direction a center turns after the snap as a potential indicator of which offensive guard is the weaker link. PFF has also created a computer program that diagrammed each of the 32,779 regular-season plays from the line of scrimmage in 2014, depicting details like wide receiver splits, depth of players off the line, motions, and route combinations. That information is then linked to a team’s video system so coaches can put eyes on what the numbers are telling them.

 

On this Friday afternoon in early January, Hornsby isn’t focusing on the minutiae of game data. He’s looking at the bigger picture (think Moneyball) and trying to enhance the process by which Pro Football Focus grades players. For pass rushers, that means factoring in things such as down and distance, how many steps the quarterback dropped back, and what position they’re rushing from. He’s also trying to establish a player’s value over a replacement player (another idea borrowed from baseball’s sabermetrics).

 

Sacks weren’t an official NFL statistic until 1982, so analytics is one department in which football lags behind other sports. But a 51-year-old bloke from England—a guy who has never played a down of football in his life—is rapidly changing the game along with a collection of far-flung employees who have only ever met through Skype.

LONDON — It’s 7 a.m. on a Friday morning in late September, and Khaled Elsayed has been up for a few hours. He awoke at 4:30 local time, right when the Washington-New York Giants Thursday Night Football tilt was ending at FedEx Field in Landover, Md. By sunrise in America, Pro Football Focus’s clients will have access to the preliminary set of information that Elsayed collects from the Giants’ 45-14 victory.

 

Elsayed, a full-time PFF analyst, works from his flat in the Stoke Newington neighborhood, above a Salvation Army and across the street from a furniture store. His fiancée asleep in the next room, he’s now breaking down plays well into the second quarter. The current play is a mere three-yard run by the Giants’ Andre Williams, but Elsayed’s stat sheet requires that he chart more than two dozen pieces of information.

 

On his first run-through, he fills in the blanks: Trick look? No. Trick play? No. Was a time out called? No. No-Huddle? No. Spot? L, for the left hash mark. Shotgun? Check. Run position? ML, for middle left of the offensive line. The runner doesn’t change course, but if he had, Elsayed would have recorded the intended point of attack and what initiated the change: F (forced by defense) or V (voluntary). Yards after contact? Two. Who made first contact? No. 31. Run concept? C, a designed cutback.

 

“One of the things that one NFL team wants us to collect is if a defender is blocked before he makes contact,” Elsayed says. “There are lots of little bits of information that might seem very useless. But NFL teams just love them, and think they’re indicative of performance.”

 

Where the ball is spotted is another piece of requested information so teams can track tendencies. There’s a different set of blanks to fill in on passing plays, including drop-back depth, time to throw, time to pressure, pass position, and turn of center—there are also checkboxes for screens, play-action, pump fakes, shotgun and pistol.

 

 

A small group of friends who met through an online forum for NFL fans in the U.K. now form the core of Pro Football Focus. “It’s a bit weird,” Elsayed says. “These guys from England helped create a new industry.”

 

 

Elsayed then rewinds the play so he can grade the offensive linemen, giving “statistics” to a position group that is generally bereft of analytical measures. He notices right away that the left guard, rookie Weston Richburg, has done a nice job. Richburg seals off defensive end Jason Jenkins, who is trying to work inside toward the ‘A’ gap. He gets a positive grade. Two of his linemates don’t fare as well. Center J.D. Walton tries to attack a linebacker at the second level but whiffs, and right guard John Jerry allows a defender to squeeze his gap. Both get a negative grade on the play. 

 

These grades sometimes draw the ire of fans, other analytics websites, NFL coaches, and even some of Pro Football Focus’s own clients. The criticism? “They take subjective perceptions and then they puree them into a number with a decimal point that looks very objective,” says Bleacher Report’s Mike Tanier, who has worked for Football Outsiders, another analytics service, since 2005. “There’s a scientific veneer there that doesn’t belong, which is worrisome.”

 

Pro Football Focus doles out grades for every snap in all of the NFL’s 267 games, assigning incremental marks from -2 to no grade to +2. (Punts are handled with a formula that former NFL punter Chris Kluwe helped devise.) To give some context, only one play in this game is graded at −2; it’s given to Washington quarterback Kirk Cousins for misreading obvious zone coverage and throwing an egregious third-quarter interception.

 

The idea is to provide a picture of how players are performing, especially when other statistics are either unavailable or misleading. An NFL fan since 1982, when the U.K.’s Channel 4 first showed American football highlights, Hornsby used to read Paul Zimmerman’s evaluations of players in Sports Illustrated that were based on film study. It inspired him to do it for every game. Hornsby’s staff works off a 106-page grading rubric; the guidelines include a 71-word definition for a “wham block” and three specific ways that an offensive lineman can receive a positive grade when double-teaming a defender.

 

When in doubt, Elsayed will flag plays for further review when the All-22 coaches’ film becomes available. “Am I being generous?” he writes after assigning a +1 grade to Eli Manning on a touchdown pass to Larry Donnell. It was a good throw, but the cornerback made the quarterback’s job easier by hesitating in coverage. On a 20-yard touchdown run by Alfred Morris, safety Antrel Rolle is only seen on the TV broadcast at the end of the play, trailing Morris. “One of the problems is, you can’t just go straight over the top—he’s got to defend the cutback,” Elsayed says. “But I need a different angle to see that. Another analyst might decide, when he looks at the All-22, that he was slow over the top. No grade for Rolle right now.”

 

 

 

 

Elsayed, 30, completes his analysis in less than six hours. He’s one of Pro Football Focus’ top analysts, having caught the football bug after catching a late-night Falcons-Saints game on TV in what he called “one drunken night at university.” Not long after, in 2007, he met Hornsby through an online forum for NFL fans in the U.K.; a small group of friends who met there now form the company’s core.

 

“It’s a bit weird,” Elsayed says. “These guys from England helped create a new industry.”

 

MADISON, Wisc. — On an NFL Sunday (or a Thursday or a Monday night), Nathan Jahnke is in his apartment two hours south of Green Bay, filling a spreadsheet with letters and acronyms that serve as a road map for every play. He works mostly in real time, creating a player participation log as the game unfolds.

 

Jahnke, who has an actuarial science degree from the University of Wisconsin-Eau Claire, memorizes shoe colors, hairstyles and even tattoos of players to quickly spot and record which 22 are on the field. A player’s play count factors into his grade in the same way that the number of yards a running back gains is only meaningful if you know how many times he carried the ball. But Jahnke is taking down more than just personnel groups. He records a player’s position based on where he lines up on the field and in relation to others.

 

 

On offense, the positions of receivers and tight ends are specified with L (left), R (right), SL (slot left), SR (slot right), SLi (slot left inside), SLo (slot left outside), and so on. If they’re on the line of scrimmage, they’re tagged with “^”. The alignment of fullbacks is marked as “L” or “R,” and in rare cases when two fullbacks are used, they’re recorded, for instance, as FBOR (outside right) and FBIR (inside right).

 

 

On defense, strong safeties are within eight yards of the line of scrimmage; free safeties are beyond that threshold. Safeties and linebackers are marked L (left), R (right) or M (middle). Three-men and four-men lines are labeled differently, and there can be both a RE (right end) and a REO (right end outside), for a player outside the tackle.

 

There are also tags for what players do on each play. A tight end that stays in to pass block is marked with a “k,” and a tight end that lines up in a two-point stance rather than a three-point stance is marked with an “a.” Any defensive back or linebacker who pass rushes is tagged with a “z,” and a defensive lineman who drops into coverage is tagged with a “v.” A defensive back that initiates contact within a yard of the line of scrimmage is tagged with a “y” to represent press coverage.

 

All NFL coaches use participation reports as guides when grading players the day after a game. But the official NFL snap count summaries aren’t always ready in time, nor is the report on the following week’s opponent from the advance scout who was sent on the road. Pro Football Focus’s data is ready by 6 a.m.

 

Jahnke, who grew up on Badgers and Packers football, answered a job ad on Pro Football Focus’s site in 2010. He scored well on a practice game, and then earned bonuses for accuracy as a part-timer before becoming full-time. “I’m doing a fourth of the league every week, so I recognize who everyone is and their tendencies,” he says. Tendencies help form the backbone of all game plans, and the weekly PDF scouting reports that Pro Football Focus gives its clients have pie charts breaking down the number of plays run per personnel group, tables of success rates based on yardage gained, bar charts showing how players are deployed and receptions by route type.

 

This year, PFF took it a step further. Jahnke spends his Tuesdays tracking advanced player participation stats from the NFL coaches’ film, such as wide receiver splits, the beginning and end points for players in motion, and the technique employed by each defensive lineman (which is represented on a 0-44 scale). Another person works on an all-route analysis, recording each receiving route run by each player throughout the game.

 

A new computer program—the play analysis tool—processes this data into individual diagrams for every play in every game. It’s sortable by personnel group, or by down and distance, or by formation, and it’s all linked to video of the corresponding play.

 

On a recent morning elsewhere in the Midwest, one veteran NFL coach was sifting through the information.

DETROIT — A day after the conference championship games, Gunther Cunningham is sitting at his desk at the Lions’ headquarters.  It’s been two weeks since Detroit lost to the Cowboys in the wild-card round, but Cunningham, a senior coaching assistant, is still dissecting the game via Pro Football Focus.

 

Cunningham is speaking over the phone to The MMQB, but he sends a picture of his oversized computer screens, on which he’s using PFF’s play analysis tool to sort defensive plays based on the Cowboys’ offensive formations.

 

A self-described “oddball” when it comes to stats and analytics, Cunningham, 68, first got a computer in 1991 to help him explore on-field tendencies when he was the Los Angeles Raiders’ linebackers coach. He started using Pro Football Focus three years ago as the Lions’ defensive coordinator; he found out about it from his friend Jon Dykema, who is staff counsel for Detroit and works with the team president on player contracts.

 

Cunningham still records some of the same statistics on his own, including time to throw or defensive targets (mostly out of habit), but he and defensive quality control coach Matt Raich lean on Pro Football Focus to shave time off their research. Before the Cowboys game, for instance, it took them less than 30 minutes to pull up all the shotgun plays Dallas had used during its previous five games. They were able to discern the personnel groups, in part, by looking at the play diagrams and corresponding video footage. “As soon as you get an idea what the team is doing, it helps with guys like Ndamukong Suh and Ezekiel Ansah, because you can say, when they’re in shotgun, pin your ears back and go get ’em,” Cunningham says. “It’s not guesswork.” If the Lions were playing New England, Cunningham adds, he could quickly sort the PFF database for all offensive plays with unbalanced lines to help prepare for the Patriots’ eligible/ineligible gambit.

 

 

“There is so much room for statistical analysis and tape analysis,” says Cunningham. “I had been thinking about investing my own money in something like this when I was done coaching.”

 

 

 

 

 

In his current role, Cunningham prepares scouting reports for the coaching staff. (A 600-page report on the Seahawks’ offense, unused after the Lions were eliminated, sits on his desk). He also helps the general manager grade players, so Cunningham often finds himself in the crosshairs of the debate that plays out in many NFL team headquarters over Pro Football Focus.

 

Vikings coach Mike Zimmer has been one of the site’s most vocal critics, calling out PFF in the opening statement of a press conference last summer. “I guarantee they don’t know who is in our blitz package and what they are supposed to do,” he said. “I would just ask everybody to take that with a grain of salt, including our fans.” Patriots coach Bill Belichick responded to a question last week about defensive sub-packages with a seeming dig, saying, “I’m sure there’s a bunch of websites that [track] that, Pro Football Extra or whatever they are.”

 

Cunningham agrees with some of Zimmer’s criticism. Even after spending five seasons as the Lions’ defensive coordinator, Cunningham found it initially difficult to wrap his head around what everyone is supposed to be doing in Teryl Austin’s new system. So PFF analysts grading players without knowing the defensive call is one area, he says, “where they are a little bit short.” But Cunningham has also seen the reverse: coaches grading favorite players more easily, or giving themselves too much credit for placing a player in the correct position when he makes a play. When one Lions player raised a stink about how Pro Football Focus graded him this season, Cunningham checked the PFF grades against his own analysis. “I wanted so much to tell him,” Cunningham says, “that they were right on the money.”

 

He views PFF analysts as young scouts, and he views their reports like those that come out of the BLESTO and National scouting organizations: more information that can help.

 

“I’ve been in the game a long time, and numbers have been really important to me,” Cunningham says. “There is so much room for statistical analysis and tape analysis. I had been thinking about investing my own money in something like this when I was done coaching. They beat me to the punch.”

Before he invested, Collinsworth was simply looking for a way to get more information on all 32 NFL teams—and faster—to better inform NBC’s national television audience of roughly 21 million. He thought he was being punked when he first heard Hornsby’s thick British accent over the phone, but PFF passed all of his tests, including a side-by-side grading of a game from 2013. Their grades differed very little, and when they did, Collinsworth says, half the time they were right and he was wrong.

 

“What really impressed me is the fact that 13 NFL teams have contracted with Pro Football Focus for their data,” Collinsworth told The MMQB last August, after becoming a business partner. “I have been around the NFL for over 30 years, I know how hard it is to get behind the wall of those teams. And they’ve got 13 teams to trust their data. That’s huge.”

 

Collinsworth’s investment enabled the company to start analyzing college football. Down the road, who knows, maybe high schools will be next? But the core of Pro Football Focus’s business remains the NFL—and coming up with new ways to help their clients find an edge and save time. In his basement Hornsby is still scribbling on the whiteboard, trying to unlock the perfect formula to better rank pass rushers. He pulls up a document on a nearby tablet that PFF has made for its teams, showing how down and distance, the quarterback’s drop-back steps, and the position from which a player is rushing directly relate to his ability to get to the QB.

 

Using their new formulas to normalize data, for instance, a predominantly right-sided rusher like Robert Quinn moves up the rankings because it’s harder to get to the quarterback from that side. A predominantly left-sided rusher, like Charles Johnson, drops a few spots. Then there’s Watt, who rushed the quarterback 669 times in 2014, and no matter how you slice it, grades twice as high as any other defender in the league.

 

Statistics, of course, don’t provide all the answers, nor do they always give new answers, but they can offer a different perspective to help break down the game.

 

“Whether a team should go for it on fourth-and-1, there’s been some analysis of that,” Hornsby says. “But the truth of it is, what is the sample size of data for that game being played in Buffalo, at a particular temperature in December, with a right guard who has a dodgy hamstring and the halfback just broke up with his girlfriend the previous day? No amount of statistics can give you that answer. Only the coach can make that decision.

 

“But what we can do, we can say to a coach, ‘If you see Calvin Johnson lining up as the inside slot receiver on a play in Week 14, and in every other circumstance where he has lined up in that position he has run this route—would that be useful to you?’ ”

 

[nj meadowlands]

I still can't get past the image of a bunch of guys watching each player on each play and assigning some sort of arbitrary "grade" to their performance on that given play.

[AFJF]

I still can't get past the image of a bunch of guys watching each player on each play and assigning some sort of arbitrary "grade" to their performance on that given play.

It does seem odd, but they're doing it well enough to get paid by NFL teams to use their info.  Grading does vary from team to team so their system may be similar to what some teams use.  I think that having 14 NFL teams use your data gives  you a bit more credibility than "a bunch of guys don't know football and have no idea what they're looking at" which is what I often see others say about their work.

[nj meadowlands]

It does seem odd, but they're doing it well enough to get paid by NFL teams to use their info.  Grading does vary from team to team so their system may be similar to what some teams use.  I think that having 14 NFL teams use your data gives  you a bit more credibility than "a bunch of guys don't know football and have no idea what they're looking at" which is what I often see others say about their work.

There certainly has to be something to it, but sometimes their grades defy the eye and common sense tests.

[BroadwayJoe12]

I'd be very interested to know how much they're paid by said NFL teams and how long the contract runs for.  Are they paying a few mere sheckles for the data dump or are they paying premium for the analysis and grade as well?  Either way, 13 teams is a sound endorsement. 

[AFJF]

 

 

I agree...but I guess that's where I'm willing to take a step back and admit that I don't know as much as I'd like to.  I have a ton of respect for what it takes to evaluate and/or coach at the pro level but with the constant information feed on the league that we get nowadays, there are a lot of fans who honestly think they know more than the pro's who will watch hundreds of hours of tape for every 2-3 hours most fans do.

[Jet9]

Eh. Baseball teams use WAR too. Hell, they made a movie out of Sabremetrics. When Billy Bean is in Cooperstown....oh, wait.

[SayNoToDMC]

[AFJF]

Eh. Baseball teams use WAR too. Hell, they made a movie out of Sabremetrics. When Billy Bean is in Cooperstown....oh, wait.

I'm not sure what sabermetrics has to do with grading actual on-field performance which is something NFL teams have been doing forever. 

[flgreen]

Because 13 NFL teams subscribe to PFF doesn't mean that they are actually giving a lot of credibility to what their analyst are saying.

 

I'm sure that some where in in the Jets PR dept  is an intern who is assigned to read the articles of Mehta and Cimini.   Probably even read the message boards.   That doesn't mean that the Jets are adhering to T0m Shane's advise.

 

Or maybe when we review some of the things the Jets have done  over the years they are.   

[AFJF]

Because 13 NFL teams subscribe to PFF doesn't mean that they are actually giving a lot of credibility to what their analyst are saying.

 

I'm sure that some where in in the Jets PR dept  is an intern who is assigned to read the articles of Mehta and Cimini.   Probably even read the message boards.   That doesn't mean that the Jets are adhering to T0m Shane's advise.

 

Or maybe when we review some of the things the Jets have done  over the years they are.   

The article mentions the fact that this data is being viewed/used by coaches/coordinators/analysts as opposed to being looked at by an intern.  That's a pretty big difference IMO.

[Jet9]

I'm not sure what sabermetrics has to do with grading actual on-field performance which is something NFL teams have been doing forever. 

You're kidding, right?

 

PFF is full of situational stats.

[AFJF]

You're kidding, right?

 

PFF is full of situational stats.

Actually I'm not.  I never got in to sabermetrics because somebody once told me that pitcher with a 5.00 ERA was super dominant but just unlucky...that was all I needed to hear.

PFF is telling me whether or not a player executed his assignment, not whether or not he failed to do it because of bad luck.

I feel like those are two different things.

[Jet9]

Actually I'm not.  I never got in to sabermetrics because somebody once told me that pitcher with a 5.00 ERA was super dominant but just unlucky...that was all I needed to hear.

PFF is telling me whether or not a player executed his assignment, not whether or not he failed to do it because of bad luck.

I feel like those are two different things.

Ok, fair enough.

 

My thing with PFF is it defies what my eyeballs see. Kyle Wilson ranked high at anything is absurd. 'But the numbers say...'. Nonsense.

[AFJF]

Ok, fair enough.

 

My thing with PFF is it defies what my eyeballs see. Kyle Wilson ranked high at anything is absurd. 'But the numbers say...'. Nonsense.

I agree on Wilson, I wouldn't argue for one second that he is possibly the worst CB ever.  However, and I've asked this before on here....can he be as bad as we all thought?  Of course your (and my) initial response would be "yes, and then some", but if he was as bad as you and I believe he was, why didn't every QB throw his way on every single play?  That seems like a bit of a stretch, but in my mind I don't recall ever seeing him make a play and you'd think if a QB threw to his man on every play he'd complete 80% of his passes.

My best guess is that much to my surprise, and yours, he must have been doing a good enough job covering his man the majority of the time, but when he blew it, he really blew it.

Even if he was as bad as we think he was, no grading system is going to be perfect and there will be some head-scratchers in there.

I have no idea how that guy is still in the NFL TBH.

[JohnnyLV]

While I do not believe in the absoluteness of the ratings I can imagine it would be useful for teams to spot under the radar players by noting "no name" players that have high ratings and then having pro player personnel check them. It is worth it in that sense just to get a macro sense of a player.

[BroadwayJoe12]

Actually I'm not.  I never got in to sabermetrics because somebody once told me that pitcher with a 5.00 ERA was super dominant but just unlucky...that was all I needed to hear.

PFF is telling me whether or not a player executed his assignment, not whether or not he failed to do it because of bad luck.

I feel like those are two different things.

That's actually a really good topic and there's some truth to that, same for guys with super low batting averages; the whole babip etc.  It usually adjusts back to their average, but it's possible. Some guys have really unlucky parts to or full seasons.