for display only
Big Blue Interactive The Corner Forum  
Back to the Corner

Archived Thread

How Much Has Scouting Accuracy Improved? (If at all...)

arcarsenal : 5/10/2018 4:34 pm
One of the slowest times of the year for football, so alas, threads like this...

Anyway. During all of the yearly draft hoopla I always wonder if scouting has actually improved over the years or if it's just always going to be the type of crapshoot we know it as. Can there be a point in the future where the percentage of successful picks (however we choose to measure that) sees a significant increase or is drafting basically just what it is?

Obviously there's no clear measure like wins or losses. I guess you could look at the percentage of drafted players who are churning out NFL careers now compared to 10 or 20+ years ago - but again, this wouldn't tell you a whole lot because the cutoff would be completely arbitrary. There are also factors like injuries that can completely alter a career path and have nothing to do with an original scouting report.

Some scouts are better than others - I think that part goes without saying.

But have technology or other data made the draft any less of a crapshoot, or is it more or less the same as it always was? Someone like Sy would probably have a good answer to this.

Are there newer strategies or evaluation methods that front offices are utilizing now that maybe they weren't in the past? Certain measurables that are historically translating to success more than others?

Drafting is such a big part of building a winner in this league, so I've always been very interested in the process and how front offices try to identify players that will have the best pro careers.

I'm sure the way the game has evolved has also impacted the things scouts look for. But the crux of this is really just me wondering if different scouting methods are yielding better results or if these avenues have been exhausted and it's always going to hinge mostly on "eye for talent," collegiate performance, and combine stuff.
Mayfield will be an interesting test for this  
giants#1 : 5/10/2018 4:40 pm : link
Not because he was #1 overall, but a lot of 'traditional' scouts liked some of the other guys (Rosen, Darnold, Allen) better because of their physical traits. Whereas the analytics pointed to Mayfield as the clear cut #1 QB prospect (IIRC, Football Outsiders has a good article on this).
Twitter  
Rocky369 : 5/10/2018 4:42 pm : link
and all the other crap that gets dug up from 20 years ago. See Josh Allen and Matt Patricia.
RE: Mayfield will be an interesting test for this  
arcarsenal : 5/10/2018 4:47 pm : link
In comment 13960548 giants#1 said:
Quote:
Not because he was #1 overall, but a lot of 'traditional' scouts liked some of the other guys (Rosen, Darnold, Allen) better because of their physical traits. Whereas the analytics pointed to Mayfield as the clear cut #1 QB prospect (IIRC, Football Outsiders has a good article on this).


Mayfield is actually a great example. I think as recently as 10-20 years ago, he'd be completely out of the conversation as the top pick because he lacks the traditional physical traits that we saw most great QB's possess for a long time.

But then guys like Drew Brees and Russell Wilson came along and seemed to change the thinking on QB height. It's more of a "concern" now than a "no way" like it may have been in the past.

I think analytics are outstanding in baseball because of how isolated every outcome is - but I've always felt like football analytics would be much harder to peg because of how many moving parts there are.

I look at football units as machines with 11 moving parts. One really bad part can drag down the rest. If most of your parts suck, your unit will probably not function well even if there's a flawless, shiny cog in there somewhere.

But I still find that there are interesting measures and I always think it's worth trying to see the game from other angles.
football analytics  
giants#1 : 5/10/2018 5:01 pm : link
From what I've seen, they are no where close to where baseball is or even where baseball was when moneyball first hit the scene.

From a prospect perspective, I think QBASE and SPARQ have shown some promise, though SPARQ looks more at measurables than performance. As for analytics that look at actual gameplay, there are some team-wide stats that show decent correlations and that smart teams could use to try and build a better roster. The biggest of these (that I'm aware of) is the correlation between yards/reception (or yards/target) and winning. I have no idea if this factored into DG's thinking, but its one of the reason's I'm excited about the Barkley pick since as good as he looks running the ball, he can really be special as a receiving back (and the Giants were dreadful there last season).

Sidenote: PFF's 'ratings' that everyone loves/hates are not analytics. They're just cumulative grades for each player after reviewing tape and grading each individual play.
.  
arcarsenal : 5/10/2018 5:08 pm : link
Yeah, I definitely don't consider PFF grades "analytics" - I think they're mildly entertaining to look at, but impossible to put any true stock into because of how wildly they vary and how inaccurate some of them seem to be.

The rookie grades they had for last season's rookies were mind boggling. Patrick Mahomes had a higher rookie grade than Evan Engram... Mahomes played one game. It made zero sense.

Stuff like QBASE I think is genuinely interesting since it's formulaic and not based on some knucklehead grading on a play by play basis like PFF does.

Mayfield's projections make me very curious - not only are they great for this class, but they're the 4th highest since 1997. Only Rivers, McNabb and Palmer had higher QBASE projections - and I'd say each of them were or are pretty good pro QB's. Palmer obviously had a litany of injury issues - but there was a time where it was Brady and Peyton as the elites, Palmer sort of in his own class right below them, and then everyone else. But these were in the Bengal days before the playoff game vs. PIT.

But seeing how Mayfield plays out is going to be intriguing... I think he's going to be pretty good. But time will tell.
this  
Eric from BBI : Admin : 5/10/2018 5:19 pm : link
is a very interesting question. I have no idea. My personal sense it hasn't gotten any better. But I may be too close to the Giants to comment on this since our drafting has tanked the past decade.
Even scouting shouldn't be analyzed in isolation  
Coach Red Beaulieu : 5/10/2018 5:23 pm : link
Take the Pats for example, their picks are more a less a bit of a cap shoot too, but they have an overall organizational philosophy, and are unique in developing and suckering other teams into trading for Cassell and Garropolo.

Wish we had that ability, but of course we haven't had a sound organizational philosophy in years
Sort of like intelligence one would imagine  
idiotsavant : 5/10/2018 6:09 pm : link
You have data and you have thinking... but you must have both.

Otherwise what conclusions do you take from your 'information'?

But. All in all, good job this year for the NY football Giants.
.  
Go Terps : 5/10/2018 6:19 pm : link
Even if it has improved with the introduction of new tools and technologies, there still figures to be that choke point at the management level that makes it difficult to determine how good or bad the scouting actually is. Where a player lands is a huge factor. RGIII for example is a punchline, but he was in the MVP (not just ROY) discussion his rookie year before he was terribly mismanaged by his owner and coach. There was another thread about what happened to Sammy Watkins. All the talent in the world, but he had the misfortune of being drafted to Buffalo and even discounting injuries he's been stuck in some bad situations.

Then there's also the possibility of a disconnect between the GM's vision and the coaching staff. We saw that right here with numerous DE/LB hybrids not working out. Maybe if Baltimore or Pittsburgh had drafted Clint Sintim he would have succeeded instead of failing here.

That leads to a question of how scouts are instructed to scout. Is their direction to simply scout talent, or is it to scout talent that they think will fit into the scheme employed by the team? I would think it's the former, where the management team makes decisions based on scouting reports that are intentionally devoid of any strategic or value judgments.

It would be cool to get Sy's or Dave Te's input here.
This might sound weird  
djm : 5/10/2018 6:25 pm : link
And I’m not even sure I believe it but I feel like scouting has likely gotten better on every level all the way down to high school. If it’s improving from the lowest levels to college it’s obviously improving in the pros. But with better scouting down low wouldn’t that sort of trickle down to the same bust rate in the pros? Better players everywhere means same hit and bust rate.

You’re never going to see a much higher hit rate. Shit is relative.
RE: .  
arcarsenal : 5/10/2018 6:53 pm : link
In comment 13960609 Go Terps said:
Quote:
Even if it has improved with the introduction of new tools and technologies, there still figures to be that choke point at the management level that makes it difficult to determine how good or bad the scouting actually is. Where a player lands is a huge factor. RGIII for example is a punchline, but he was in the MVP (not just ROY) discussion his rookie year before he was terribly mismanaged by his owner and coach. There was another thread about what happened to Sammy Watkins. All the talent in the world, but he had the misfortune of being drafted to Buffalo and even discounting injuries he's been stuck in some bad situations.

Then there's also the possibility of a disconnect between the GM's vision and the coaching staff. We saw that right here with numerous DE/LB hybrids not working out. Maybe if Baltimore or Pittsburgh had drafted Clint Sintim he would have succeeded instead of failing here.

That leads to a question of how scouts are instructed to scout. Is their direction to simply scout talent, or is it to scout talent that they think will fit into the scheme employed by the team? I would think it's the former, where the management team makes decisions based on scouting reports that are intentionally devoid of any strategic or value judgments.

It would be cool to get Sy's or Dave Te's input here.


All good points and stuff that no doubt impacts the overall result of the player.

Scouting obviously only gets the player to the team if they like him enough, but you're right that once he arrives, the coaching staff has a major impact on where the prospect goes from there.

Scheme fit also plays a major role. Players who are miscast are often doomed to fail unless their talent finds a way to supersede it.

I'd definitely like to know how the overall approach has evolved and changed over the years and how much a current staff and the things they employ impact the way a front office grades a player.

Bradley Chubb, for example, is a guy who, as far as I saw, was strictly a 4-3 LDE/RDE @ NC St. So, if you're a team that employs a 3-4, how much does that change the way you view the player? Theoretically, it should have a pretty significant impact on a decision to draft him or not - but I wonder if some execs just value the player's talent to the point where they say "we're going to take this guy anyway, find a way to make it work."
Agree with Terps  
UConn4523 : 5/10/2018 7:00 pm : link
simply too many variables to factor in. I’m guessing scouts are good at their jobs but there’s too many unknowns, many previously mentioned. Tom Brady may very well be a nobody if the Browns selected him. Chance and circumstance don’t factor into scouting.
I feel the scheme thing is overplayed  
jcn56 : 5/10/2018 7:01 pm : link
Look at the better coaches in the league, and they find a way to make players work in their scheme, not to find players who fit their scheme (BB being the obvious reference).

I wonder how often teams go back and analyze their scouting. Do they self-scout themselves, go back and revisit their draft grades 2-3 years after the fact, to see where they were right and where they were wrong? Do they adjust accordingly?

This is why I pause while others celebrate our draft - we're using the same scouts and a couple of guys different at the top. Now maybe those two guys make all the difference, but how likely is it that the results are markedly better when most of the scouting is still coming from the same core?
I suspect there are less hidden gems  
George from PA : 5/10/2018 7:09 pm : link
But accuracy, I doubt it is any better.
Maybe the scouts have been fantastic the past few years  
Daniel in Kentucky : 5/10/2018 7:12 pm : link
Getteleman would know more than any of us having worked there for years.

I think firing Marc Ross almost immediately after he arrived in New York says a lot. I wasn’t impressed with Ross after watching that show Finding Giants. I was like, “Really!?!?” You are a significant part of our draft selection process!

Maybe the blame lies with Ross and Reese.

That is my suspicion at least.

Loved this past draft!

jcn  
Go Terps : 5/10/2018 7:15 pm : link
My hunch is that both McAdoo and Coughlin (and their staffs) were a problem when it came to player development and incorporation into the game plan. We had some major blind spots even in Coughlin's best days. I don't know when the last time was that we actually drafted a linebacker and turned him into a good player. It might actually be Jessie Armstead, I don't know.

And have we developed a quarterback not named Eli Manning into anything since Jeff Hostetler?

But I completely agree about the best coaches modifying their approach to fit the strengths of the players. I heard a great quote in a history podcast recently: "Amateurs deal in tactics, masters deal in logistics." I think that applies here.

A troubling thought already is that what we know as Shurmur's offense doesn't really align with what we know to be Eli's strengths. Hopefully Shurmur recognizes that.
JCN  
UConn4523 : 5/10/2018 7:18 pm : link
there’s no way to know what’s scouting and what’s any other variable.
.  
arcarsenal : 5/10/2018 7:25 pm : link
Chance and circumstance definitely don't factor into scouting - nor can injuries to players without an injury history.

So, of course, what happens under the tutelage of the coaches and on the field will have a massive impact on how the player develops.

That said - we've also seen countless players in this league struggle with their original team and not be any better in other systems with different sets of coaches. And sometimes, these players were tremendously productive in college.

There won't be a blanket reason for every failed prospect - but I do wonder where the disconnect lies when a player is so highly regarded in a draft and then never amounts to anything in the NFL. With the amount of prepwork and scouting that is done on these guys, it always sort of surprises me when "can't miss" guys like Gallery or Richardson bust. With some guys, it's self-inflicted (a-la Ryan Leaf) - but otherwise, I always wonder what it is that was missed on these guys or why the NFL production wound up so far off the projection.
My impression  
WillVAB : 5/10/2018 7:27 pm : link
In some ways yes, in other ways no.

30 years ago a guy could get drafted that half the teams never heard of. Now everyone knows at least something about all of the guys in the draft pool.

Technological advances and the amount of information out there has to have made it easier for scouts to do their job. Teams invest more money and resources into it than they did years ago.

I suspect the challenges teams experience now are the same as they’ve always been on a certain level. Some scouts/coaches/mgmt may have a bias towards production, others may be biased towards measurables/athleticism. Teams still struggle to project players into their specific system. Management/scouts may butt heads on specific players or general team building philosophy. Maybe certain politics are at play within a team where the decision makers value one scout’s opinion over another’s. Maybe a team picks for need or over values certain positions and ignores the scout’s analysis on a player. Maybe a team has a sound process but just got unlucky in some way. The list goes on and on.
Scouting can be good or bad, but . . . .  
TC : 5/10/2018 7:36 pm : link
the rookies still need to go out on the field. And irrespective how well they're scouted, measured or performed in school, a lot of it is still up to attitude, and being in the right place. A bad attitude, or bad coach or being made a square peg in a round hole can take a player with innate ability and turn him into crap. And the opposite is also true.
Arc  
UConn4523 : 5/10/2018 7:44 pm : link
I think a lot of that is due to a player basically being damaged goods spending 3/4 years with a team, failing, then not getting reps, then becoming expensive to sign since they would have vet status. If you stink for 4 years why take a flier on you with a guaranteed contract when I can get a younger, cheaper player with more upside?
RE: Arc  
Optimus-NY : 5/10/2018 9:57 pm : link
In comment 13960669 UConn4523 said:
Quote:
I think a lot of that is due to a player basically being damaged goods spending 3/4 years with a team, failing, then not getting reps, then becoming expensive to sign since they would have vet status. If you stink for 4 years why take a flier on you with a guaranteed contract when I can get a younger, cheaper player with more upside?


Precisely.
Research and time spent by experienced experts  
giantsFC : 5/10/2018 10:40 pm : link
Has vastly improved now, but at the end of the day I it’s still a subjective opinion based on observations a team is relying on. And nobody can predict the future.
But I think the more prepared, modernized and experienced staff can help eliminate completely epic fail drafts like the giants have had of recent. But there will still be plenty of players who don’t pan out for what er reasons every draft.
RE: JCN  
jcn56 : 5/10/2018 11:46 pm : link
In comment 13960648 UConn4523 said:
Quote:
there’s no way to know what’s scouting and what’s any other variable.


I don't agree with that at all - and the fact that these teams sink major resources into scouting should tell you that they don't either.

After the fact, a team can definitely go back and review their scouting of a class to see how effective it was. Sure, it's not going to be 100% accurate - some players will be affected by situation and injury to an extent you can't predict.

But for other players - in particular, guys that the Giants have downgraded for one reason or another - it should be relatively easy for them to look back and see whether they were on point. Compare the scouting report to the first 3-4 years of that player's career, and see whether or not the shortcomings that were identified in the report were present on the field.
Predicting any human behavior  
Daniel in MI : 5/11/2018 9:38 am : link
Is amazingly complex. Football is a complex behavior making it harder. Now in addition to that you have what statisticians in this area call “range restriction.” That is, if you think of a bell curve of all college players, now you’ve lopped off the bottom 85% (at least) of the performance curve that don’t even get considered. So you’re trying to work on just the top little end of the curve. If you were dealing with the whole range as is usually the case, I bet scouts would be very good at predicting which players would be better than others. Joe Schmo who started 3 games at RB for NE Missouri A&M is probably not as good a prospect as Barkley, and the scouts and combine would show that in no uncertain terms. But take all top prospects, they are all big, strong, fast, etc. They have already eliminated all those people, so now you have only players that were at least excellent in college, and you’re trying to discriminate among those talented people.

Add to that system, motivation, coaching, who else is on the team (poor Headen), injury, rule changes, game evolution, etc. and it’s a very very difficult task. I think we already do a good job of picking prospects from among all college players. Trying to sort all those variables becomes almost impossible.
RE: jcn  
giants#1 : 5/11/2018 9:55 am : link
In comment 13960647 Go Terps said:
Quote:
My hunch is that both McAdoo and Coughlin (and their staffs) were a problem when it came to player development and incorporation into the game plan. We had some major blind spots even in Coughlin's best days. I don't know when the last time was that we actually drafted a linebacker and turned him into a good player. It might actually be Jessie Armstead, I don't know.

And have we developed a quarterback not named Eli Manning into anything since Jeff Hostetler?

But I completely agree about the best coaches modifying their approach to fit the strengths of the players. I heard a great quote in a history podcast recently: "Amateurs deal in tactics, masters deal in logistics." I think that applies here.

A troubling thought already is that what we know as Shurmur's offense doesn't really align with what we know to be Eli's strengths. Hopefully Shurmur recognizes that.


Kennard was solid, especially considering where he was drafted (6th?). Goff was also looking good before the injuries took their toll. The biggest problem with the LBs is that, aside from Sintim and Gerris Wilkerson, we haven't drafted any in the top 3 rounds in 20+ years.

QBs - Eli's been the starter since 2004, that kind of negates investing significant resources there. How often do mid-round picks really become solid starters. And outside of Webb/Lauletta, who it's obviously still early for, we've drafted Nassib (4th), Bomar (5th) and Woodson (6th). The lack of development from Nassib was a disappointment, but the other's were definite wildcards.

That said, the development of OL is probably the biggest area the past two regimes struggled. Despite many claims, Reese invested a lot of draft capital in the OL and had very little to show for it whether it was high picks (Flowers, Pugh, Richburg), or mid and late round picks (Hart, Bisnowaty, Herman, Mosley, McCants, Brewer, Petrus, Koets). Beatty was arguably the only *hit* since Reese took over, though Pugh's biggest issue was injuries, not talent or development.

As for your point about Shurmur, what is it that you don't like? His Vikings stint absolutely shows he adapts to a QBs strengths as in 2016 the Vikings offense featured a lot of short, quick throws with the accurate Bradford at QB and the porous OL and in 2017 after Bradford's injury they completely revamped things to rely more on moving the pocket and play action, which Keenum was more adept at.
RE: RE: JCN  
UConn4523 : 5/11/2018 12:30 pm : link
In comment 13960840 jcn56 said:
Quote:
In comment 13960648 UConn4523 said:


Quote:


there’s no way to know what’s scouting and what’s any other variable.



I don't agree with that at all - and the fact that these teams sink major resources into scouting should tell you that they don't either.

After the fact, a team can definitely go back and review their scouting of a class to see how effective it was. Sure, it's not going to be 100% accurate - some players will be affected by situation and injury to an extent you can't predict.

But for other players - in particular, guys that the Giants have downgraded for one reason or another - it should be relatively easy for them to look back and see whether they were on point. Compare the scouting report to the first 3-4 years of that player's career, and see whether or not the shortcomings that were identified in the report were present on the field.


I will think you are putting entirely too much on the shoulders of talent evaluators. There’s are simply too many other variables that don’t show up on tape or in the games and practices from which they are getting data. And like the NFL and the amount they spend on QBs, spending a ton of money on scouting doesn’t always mean it’s a wise decision or the best decision. You spend money and hope it’s money well spent, that your guys will find an edge somehow. The reason they don’t, IMO, it’s these unknowns that will always be there and continue to grow.

I mean you’ve got David Price making $31 million per year in the MLB talking about how Fortnite and video games are a distraction. How do you account for stupid shit like that with scouting? How can you predict when a behavior never on display in tape now becomes a factor?
And going back 3-4 years to see what went wrong  
UConn4523 : 5/11/2018 12:34 pm : link
still doesn’t even address coaching, personal life, etc. How do you quantity those things and more importantly, how do you apply that when coaches change, teammates change, etc?

Scouting in the NFL is always going to be heavily reliant on factors outside of a scouts control. There’s simply no way to account for a lot of it all things off the football field.
Of course it's inexact - nobody expects it to be  
jcn56 : 5/11/2018 9:34 pm : link
But even if they're only 50% right at best - there's a big difference between 50% and 35%.

They can very easily go back and self scout years later, and look for instances where they documented physical tendencies, whether good or bad, and see how effective they were and how much those tendencies actually impacted their game.
Back to the Corner