Big Blue Interactive The Corner Forum  
Back to the Corner
 

Archived Thread

NFT: The Tyranny of Metrics

baadbill : 5/17/2018 10:46 am
A new book in which the author points out that society's increasing use of metrics to evaluate performance has consequences for the very behavior being measured; consequences that are sometimes good and sometimes bad.

As an example, the "performance" of surgeons is increasingly measured by metrics that count surgical success (avoiding death or negative outcomes). The result? Studies reveal that, increasingly, surgeons decline to offer surgery to patients with complex conditions and questionable outcomes, resulting in deaths because their conditions, while requiring surgery, involve outcomes that are not "safe" enough from the perspective of surgeons engaged in "risk avoidance" because they are now being measured by metrics.

The author says he is not against the use of metrics; he is only against the overuse of metrics; against metrics not being a tool for judgment, but rather metrics being a complete replacement for judgment (judgment based upon experience).

It got me to wondering: Has the increased reliance upon metrics in sports had unintended, unforeseen, negative impacts that has changed how baseball (or other sports) is played?
Link - ( New Window )
Metrics nerds name my skin crawl  
Coach Red Beaulieu : 5/17/2018 10:59 am : link
Metrics-centric goals are stupid. Puked in my mouth a little when the org touted a 70% completion goal for Eli.

That's not your F***ING goal, your goal is to make the damn playoff and win the damn super bowl!
Agree 200%  
Sec 103 : 5/17/2018 11:00 am : link
People are reacting to making metrics look good instead of doing the right thing.....
I read this book because it applies to my field  
jcn56 : 5/17/2018 11:01 am : link
but I thought it was somewhat of an oversimplification.

Is there too much of a reliance on metrics? Sometimes; sometimes what can't be measured should count more than what can be measured. That's the subjectivity that has always come into play whenever someone tries to combine math and science to analyze a subject matter that's inexact.

I didn't think much of sports while I was reading it. The only sport where metrics really have that strong a hold is baseball, and the argument can be made that they've only benefited those teams with a heavy reliance on metrics. My immediate thought was to things like the balanced scorecard in businesses or the advanced metrics used in finance or IT, where you sometimes have analytics that fly in the face of accepted practice/rule of thumb, and you have to judge whether to go by convention or follow the numbers.
RE: I read this book because it applies to my field  
baadbill : 5/17/2018 11:15 am : link
In comment 13966486 jcn56 said:
Quote:
but I thought it was somewhat of an oversimplification.

Is there too much of a reliance on metrics? Sometimes; sometimes what can't be measured should count more than what can be measured. That's the subjectivity that has always come into play whenever someone tries to combine math and science to analyze a subject matter that's inexact.

I didn't think much of sports while I was reading it. The only sport where metrics really have that strong a hold is baseball, and the argument can be made that they've only benefited those teams with a heavy reliance on metrics. My immediate thought was to things like the balanced scorecard in businesses or the advanced metrics used in finance or IT, where you sometimes have analytics that fly in the face of accepted practice/rule of thumb, and you have to judge whether to go by convention or follow the numbers.


While you may not agree with its conclusions, was it worth the read? Was it a chore?
It was an easy enough read, so I don't think I'd call it a chore.  
jcn56 : 5/17/2018 11:26 am : link
Odd thing about the book - it might as well have been written 20 years ago, as far as I'm concerned. There's not a passing mention of big data anywhere in the book (and that's partly the field I'm in). IMO, you're missing a pretty important topic if you don't consider the impact that big data and the availability of cheap, large scale analytics has had on some of these companies.

It's an interesting read for an overview, and basically focuses on the overuse of metrics in management. I'm more concerned with the use of metrics and analytics for tactical decisions than trying to grade performance based on key performance indicators. His book primarily looks at the impact on doctors, teachers, etc. when metrics are overused in an attempt to evaluate them.

I don't regret reading it, but I walked away thinking that it wasn't what I was expecting (and I had preordered the book based on an Amazon recommendation, before any reviews were really available).
Obviously  
giantsfan44ab : 5/17/2018 11:29 am : link
the outcomes of sports is a relative non-issue to metrics being used in the medical field. But the whole issue of malpractice lawsuits is another major factor to leading rejecting patients.

As for sports, casual fans can sit and complain about it as much as possible but the simple fact is your favorite team is spending tens of millions of dollars supporting analytics departments and rely on them heavily in the decision making process. Obviously the dumb stats they show on the broadcasts are more for entertainment value than for driving decisions. The amount its used varies by sport and by team.

Basbeall is an easy target for analytics as the game is broken down into individualized events. It's becoming pretty easy to value how much a player brings to the table as one doesn't affect the performance of another player (aside from catchers and pitcher interactions). Baseball and football are harder to isolate "true" performance but player tracking data is helping this in those sports.
And in that sense, I wonder whether we haven't seen the last  
jcn56 : 5/17/2018 11:30 am : link
of expanding use of analytics in baseball. I wonder if they don't start gathering data from these ball/strike location systems, and start to use them when making managerial decisions during the game (something along the lines of real-time scouting). Obviously, this is being done to some degree just by eye during games, but I wonder if we don't see it go full on streaming data analysis.
I believe the Rockets GM Morey is a big analytics (metrics) guy  
giants#1 : 5/17/2018 11:36 am : link
and it's helped them get the #1 seed this year.

Also, the whole 3 pt "craze" in the NBA is due to metrics. The teams that have moved towards more 3s at the expense of less long 2s have definitely benefited.

As for the NFL - I can see a few areas where metrics can challenge traditional thinking:
- More passing: there's some research that shows higher yards/reception or even better yards/attempt (or lower opponents yards/rec) leads to more wins. Naturally, this means that teams should emphasize passing more than rushing (not necessarily deep passing since the comp % drops there). Would this increase enjoyment? Depends on if you like scoring or more traditional smash mouth football.

- Going for 2 pts: Tomlin and the Steelers are the most aggressive here. Not sure if it's translated to more wins, but (IMO) its definitely more exciting than an XP

- Punting less: anecdotally teams are getting slightly more aggressive and going for it more often if they've crossed the 50. Not sure if the more aggressive teams have statistically better records or not, but this would definitely increase excitement (punts are one of the most boring plays in the game). The real key will be to see which, if any, teams start going for it on their own side of the field. Not necessarily deep in their territory, but if you have a 4th and short around your own 35-40 yard line it's interesting and I believe the analytics say to go for it. Doug Pederson was as aggressive as anyone last year and now with some SB capital in his back pocket, it'll be interesting to see if he gets even more aggressive in situations like these.
RE: I believe the Rockets GM Morey is a big analytics (metrics) guy  
BigBlueDownTheShore : 5/17/2018 11:41 am : link
In comment 13966534 giants#1 said:
Quote:
and it's helped them get the #1 seed this year.

Also, the whole 3 pt "craze" in the NBA is due to metrics. The teams that have moved towards more 3s at the expense of less long 2s have definitely benefited.

As for the NFL - I can see a few areas where metrics can challenge traditional thinking:
- More passing: there's some research that shows higher yards/reception or even better yards/attempt (or lower opponents yards/rec) leads to more wins. Naturally, this means that teams should emphasize passing more than rushing (not necessarily deep passing since the comp % drops there). Would this increase enjoyment? Depends on if you like scoring or more traditional smash mouth football.

- Going for 2 pts: Tomlin and the Steelers are the most aggressive here. Not sure if it's translated to more wins, but (IMO) its definitely more exciting than an XP

- Punting less: anecdotally teams are getting slightly more aggressive and going for it more often if they've crossed the 50. Not sure if the more aggressive teams have statistically better records or not, but this would definitely increase excitement (punts are one of the most boring plays in the game). The real key will be to see which, if any, teams start going for it on their own side of the field. Not necessarily deep in their territory, but if you have a 4th and short around your own 35-40 yard line it's interesting and I believe the analytics say to go for it. Doug Pederson was as aggressive as anyone last year and now with some SB capital in his back pocket, it'll be interesting to see if he gets even more aggressive in situations like these.


Its more prevalent in high school football then in higher levels of football. There was some football team that always went for it on 4th down, and always did on-side kickoffs, simply because the odds at that level said that there was no benefit to trying to flip the field. I think the team won multiple championships doing it as well.
I think the NFL might be making a turn towards expanding their use  
jcn56 : 5/17/2018 11:44 am : link
It seems that their adoption of services like PFF had more to do with snap counts and other metrics that they track than any kind of ratings. When you see that teams are paying money to get this data collected and structured, it tells you what direction they're heading.
Found the team  
BigBlueDownTheShore : 5/17/2018 11:46 am : link
Here is a link to an article on him.

The coach is 77-17 in 2015.


The highly successful high school coach who never punts has another radical idea - ( New Window )
This is particularly a HUGE problem in Education....  
Britt in VA : 5/17/2018 11:50 am : link
thanks for the recommendation.
RE: This is particularly a HUGE problem in Education....  
baadbill : 5/17/2018 12:00 pm : link
In comment 13966557 Britt in VA said:
Quote:
thanks for the recommendation.


The book review mentions educators being measured by test scores adopting curricula designed to "test well"
Yeah, he covers both education and medicine and the ethical  
jcn56 : 5/17/2018 12:03 pm : link
dilemma associated with taking a metrics-based management approach in both of those fields, and he does a pretty good job.

What he does sort of gloss over is that the metrics based push in education is mostly due to the billion dollar testing business and all their lobbying. If all that testing was made free overnight, somehow I don't think people are as gung-ho about metrics any more.
RE: Yeah, he covers both education and medicine and the ethical  
Britt in VA : 5/17/2018 12:03 pm : link
In comment 13966568 jcn56 said:
Quote:
dilemma associated with taking a metrics-based management approach in both of those fields, and he does a pretty good job.

What he does sort of gloss over is that the metrics based push in education is mostly due to the billion dollar testing business and all their lobbying. If all that testing was made free overnight, somehow I don't think people are as gung-ho about metrics any more.


Yeah, it's all really messed up. This is a must read for me.
It's evolution ...  
DonQuixote : 5/17/2018 12:07 pm : link
... people will use the metrics to gain some advantage, then if those are uniformly adopted, some other angle will be used to gain an advantage over the standard metrics, etc. It's the same thing on Wall Street and in evolutionary biology. Current practices set up a certain environment for future innovations to succeed. If the current metrics approach has gaps, they will be exploited.

A poster above compared this to standardized testing, which is a bit off. In baseball, your outcomes are clearly defined by wins. In education, standardized tests are an inexact way to measure outcomes, so testing to the test may not achieve anything. If high test scores did achieve something significant, then nobody would have a problem with trying to achieve high test scores.

The bottom line is that you can't really be against any approach unless you have a better approach to replace it.

He also mentions the ethical dilemma associated with the medical  
jcn56 : 5/17/2018 12:08 pm : link
industry, but doesn't consider that even without metrics, that industry is driven mostly by profits and not by patient care. At the end of the day, the primary focus would still be to get costs down and revenues up.
The point about surgeons is on the money  
SomeFan : 5/17/2018 12:10 pm : link
It is concerning.
I saw it in corporate legal ... measuring performance on absolutely  
baadbill : 5/17/2018 12:10 pm : link
idiotic stuff that had immediate ramifications... when people's compensation is measured via metrics, it is to be expected you will get exactly what you ask for; you are setting stupid goals and your employees will excel at achieving those metrics (i.e. gaming the system).

What everyone with expertise in their "area" sees, is that the corporate bean counters are only concerned with the fact that they are counting something ... they aren't interested in dealing with things that can't be counted (i.e. the very "expertise" itself that makes the job worth doing in the first place).
The issue is  
giantsfan44ab : 5/17/2018 12:17 pm : link
picking metrics blindly without consistently re-evaluating them to see if they align with the actual goals that are trying to be attained.
And I think teachers are a perfect example...  
baadbill : 5/17/2018 12:17 pm : link
The 2 or 3 teachers in my lifetime that had the greatest effect upon me - upon my entire life - were the ones that caused me to "love to think"... as such, it wasn't so much the content they were teaching, it was "how" they taught... it was how they changed how my brain was wired ... they were, simply put, great teachers... and I'm not sure how that gets measured by metrics
RE: And I think teachers are a perfect example...  
giants#1 : 5/17/2018 12:20 pm : link
In comment 13966597 baadbill said:
Quote:
The 2 or 3 teachers in my lifetime that had the greatest effect upon me - upon my entire life - were the ones that caused me to "love to think"... as such, it wasn't so much the content they were teaching, it was "how" they taught... it was how they changed how my brain was wired ... they were, simply put, great teachers... and I'm not sure how that gets measured by metrics


Something like a student satisfaction survey would capture that.
RE: I saw it in corporate legal ... measuring performance on absolutely  
bluepepper : 5/17/2018 12:22 pm : link
In comment 13966585 baadbill said:
Quote:

What everyone with expertise in their "area" sees, is that the corporate bean counters are only concerned with the fact that they are counting something ... they aren't interested in dealing with things that can't be counted (i.e. the very "expertise" itself that makes the job worth doing in the first place).

This is called the McNamara Fallacy after the SecDef during Vietnam.

The Best and the Brightest is highly recommended if you haven't read it. A take down of guys like McNamara and McGeorge Bundy who thought they had it all figured out.
Link - ( New Window )
RE: RE: And I think teachers are a perfect example...  
baadbill : 5/17/2018 12:23 pm : link
In comment 13966601 giants#1 said:
Quote:
In comment 13966597 baadbill said:


Quote:


The 2 or 3 teachers in my lifetime that had the greatest effect upon me - upon my entire life - were the ones that caused me to "love to think"... as such, it wasn't so much the content they were teaching, it was "how" they taught... it was how they changed how my brain was wired ... they were, simply put, great teachers... and I'm not sure how that gets measured by metrics



Something like a student satisfaction survey would capture that.


Maybe, but I doubt it. It would capture popularity for sure, but not necessarily "how to think" (moreover, I'm not sure I realized the impact those teachers had on me until years later).
I know I've started several threads on this through the years....  
Britt in VA : 5/17/2018 12:27 pm : link
quick search found this one from a couple years ago.

Link - ( New Window )
a (highly) flawed test doesn't mean metrics are bad  
giants#1 : 5/17/2018 12:33 pm : link
or not useful. It just means that test (metric) is useless and/or poorly designed.
Regarding Metrics, A Couple Of Phrases I Come Across At Work  
Trainmaster : 5/17/2018 12:34 pm : link
“Bad data is worse than no data”

“Measure what is important, not what is easy to measure”

“In God we trust; all others bring data”


And a few contadictory ones read first in Carl Sagan books from years ago (I don’t think he originated them and he didn’t claim to have):

“If you can’t measure it, it doesn’t exist”

“Absence of evidence, isn’t evidence of absence”

“The burden of proof is on those making extraordinary claims”




RE: This is particularly a HUGE problem in Education....  
BigBlueDownTheShore : 5/17/2018 12:46 pm : link
In comment 13966557 Britt in VA said:
Quote:
thanks for the recommendation.


The no child left behind act really goes to this point. The thought process is a good one, but taking away kids electives to force them into basically a test passing course is doing more harm then good. Being able to do electives, helps kids figure out what they actually like and could excel at.
RE: Agree 200%  
Jim in Fairfax : 5/17/2018 12:53 pm : link
In comment 13966483 Sec 103 said:
Quote:
People are reacting to making metrics look good instead of doing the right thing.....

According to the metrics, the maximum you can agree is 100%.
Its invaded sports as well.  
Brown Recluse : 5/17/2018 12:59 pm : link
.
Don't mind me.  
Brown Recluse : 5/17/2018 1:07 pm : link
I didn't read the OP as well as I should have.
RE: RE: I saw it in corporate legal ... measuring performance on absolutely  
jcn56 : 5/17/2018 2:54 pm : link
In comment 13966603 bluepepper said:
Quote:
In comment 13966585 baadbill said:


Quote:



What everyone with expertise in their "area" sees, is that the corporate bean counters are only concerned with the fact that they are counting something ... they aren't interested in dealing with things that can't be counted (i.e. the very "expertise" itself that makes the job worth doing in the first place).


This is called the McNamara Fallacy after the SecDef during Vietnam.

The Best and the Brightest is highly recommended if you haven't read it. A take down of guys like McNamara and McGeorge Bundy who thought they had it all figured out. Link - ( New Window )


The worst part of it was history repeating itself with Rumsfeld's attempt at a metrics approach based conflict.

In both cases, though, it was more flawed policy with an attempt to spin with metrics than the other way around, IMO.
RE: RE: Agree 200%  
Sec 103 : 5/17/2018 3:14 pm : link
In comment 13966627 Jim in Fairfax said:
Quote:
In comment 13966483 Sec 103 said:


Quote:


People are reacting to making metrics look good instead of doing the right thing.....


According to the metrics, the maximum you can agree is 100%.

Touche
With kids in school those metrics loom large  
idiotsavant : 5/17/2018 3:19 pm : link
Seems like when when we were kids you had mid term and final, maybe one project and maybe a 10% swing measure of class attitude.

Today as many as 80 gradeable items per semester per class seems like the order of the day. With 4 core subjects, two electives plus gym, a language and art. That's 9.

So the metric is compliance and ones ability or willingness to manage a busy schedule moreso than math, reading, writing, science and history.
RE: RE: I saw it in corporate legal ... measuring performance on absolutely  
baadbill : 5/17/2018 3:19 pm : link
In comment 13966603 bluepepper said:
Quote:
In comment 13966585 baadbill said:


Quote:



What everyone with expertise in their "area" sees, is that the corporate bean counters are only concerned with the fact that they are counting something ... they aren't interested in dealing with things that can't be counted (i.e. the very "expertise" itself that makes the job worth doing in the first place).


This is called the McNamara Fallacy after the SecDef during Vietnam.

The Best and the Brightest is highly recommended if you haven't read it. A take down of guys like McNamara and McGeorge Bundy who thought they had it all figured out. Link - ( New Window )


Thanks. I've heard of the book before - the classic on the war... I've ordered the book but I'm not sure I'm happy about it; I'm afraid I'll find it depressing... at my age I lean heavily towards uplifting … haha
RE: I read this book because it applies to my field  
BigBlueDownTheShore : 5/17/2018 3:38 pm : link
In comment 13966486 jcn56 said:
Quote:
but I thought it was somewhat of an oversimplification.

Is there too much of a reliance on metrics? Sometimes; sometimes what can't be measured should count more than what can be measured. That's the subjectivity that has always come into play whenever someone tries to combine math and science to analyze a subject matter that's inexact.

I didn't think much of sports while I was reading it. The only sport where metrics really have that strong a hold is baseball, and the argument can be made that they've only benefited those teams with a heavy reliance on metrics. My immediate thought was to things like the balanced scorecard in businesses or the advanced metrics used in finance or IT, where you sometimes have analytics that fly in the face of accepted practice/rule of thumb, and you have to judge whether to go by convention or follow the numbers.


Here is a good book for you on big data. An excellent book that discusses big data and its implications. Not really sports-focused, but a great read nonetheless.
Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are - ( New Window )
RE: RE: RE: And I think teachers are a perfect example...  
DonQuixote : 5/17/2018 4:07 pm : link
In comment 13966607 baadbill said:
Quote:
In comment 13966601 giants#1 said:


Quote:


In comment 13966597 baadbill said:


Quote:


The 2 or 3 teachers in my lifetime that had the greatest effect upon me - upon my entire life - were the ones that caused me to "love to think"... as such, it wasn't so much the content they were teaching, it was "how" they taught... it was how they changed how my brain was wired ... they were, simply put, great teachers... and I'm not sure how that gets measured by metrics



Something like a student satisfaction survey would capture that.



Maybe, but I doubt it. It would capture popularity for sure, but not necessarily "how to think" (moreover, I'm not sure I realized the impact those teachers had on me until years later).


It would be interesting to have some form of evaluations done by students ten years out or something.
RE: Regarding Metrics, A Couple Of Phrases I Come Across At Work  
DonQuixote : 5/17/2018 4:08 pm : link
In comment 13966614 Trainmaster said:
Quote:


“The burden of proof is on those making extraordinary claims”




Isn't this one something like "extraordinary claims require extraordinary evidence"?
RE: RE: I read this book because it applies to my field  
jcn56 : 5/18/2018 12:06 am : link
In comment 13966816 BigBlueDownTheShore said:
Quote:
In comment 13966486 jcn56 said:


Quote:


but I thought it was somewhat of an oversimplification.

Is there too much of a reliance on metrics? Sometimes; sometimes what can't be measured should count more than what can be measured. That's the subjectivity that has always come into play whenever someone tries to combine math and science to analyze a subject matter that's inexact.

I didn't think much of sports while I was reading it. The only sport where metrics really have that strong a hold is baseball, and the argument can be made that they've only benefited those teams with a heavy reliance on metrics. My immediate thought was to things like the balanced scorecard in businesses or the advanced metrics used in finance or IT, where you sometimes have analytics that fly in the face of accepted practice/rule of thumb, and you have to judge whether to go by convention or follow the numbers.



Here is a good book for you on big data. An excellent book that discusses big data and its implications. Not really sports-focused, but a great read nonetheless. Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are - ( New Window )


Yup - I've read that one but I agree, that's a good one, worthy of a read in particular after this whole Cambridge Analytics/Facebook mess.

How people still use those sites after all that is beyond me.
Data is neutral  
idiotsavant : 5/18/2018 9:38 am : link
It's a tool.

And some people will use it wrong (utopian modernists?, panglossians? Malthusians? Totalitarians?)

While others take it for what it's worth - and no more.

It's when applied to human behavior, ideas that we can 'formula' 'solutions' are very scary ...and should be.

Applied to sports, fine, but even then without common sense and without blending with other factors.... No good.

Anything new, people will over reach, over use, and power grab. Always.

Darwin? Great.

Darwin's (son, nephew?, the eugenecist?) Horrible.

Etc.

Back to the Corner