A key thing worth mentioning is the ample evidence that some people are much better at forecasting than others. And I understand it, these "superforecasters" are not typically relying on ultra-fancy mathematical models with minimal assumptions. They use vibes, numbers, and some arithmetic here and there. (The difference is that their vibes are of better quality than most people's.)
That's right. Research shows that there are "superforecasters" who are significantly and consistently more accurate than other forecasters (I was one of the forecasters identified as a "superforecaster" in the original Good Judgment Project tournament). While these skilled forecasters often use models to make forecasts, they don't rely on any single method outside of using their judgment about what they vibes indicate. As you say, the difference is that their judgment about the vibes is generally good.
Yeah I don't know how to have cut through with this concept but I think about it a lot. Vibeness seem orthogonal to how true something is. Some vibes are true, some robust models are false.
I guess I'd say that pretty much all models are false—in the sense that the simplify away features of reality—but some are useful. I think the same can be said of some interpretations of the vibes. In each case the real question is, do they yield accurate predictions?
Whether I am reading reviews to purchase a software or the next "health hack" in news media or election forcaster (like nate silver), I always puzzle about how to trust a piece of information.
Do you use any framework or heuristic for assessing a piece of information ?
For example, lets take a health hack, Resistant starch consumption as a means for glucose regulation. How do you frame/assess following parts of the context
- context of information ( social media feed, news article in mainstream media, post on less mainstream media , group discussion with actual humans, group discussion with virtual avatars , apnews , medical journal )
- message itself ( resistant starch is better for your health )
- benefits and cost of accepting message (this is personal i get it)
- platform ( ad somewhere, social media feed, mass media ) and their incentives
- author ( a reporter ? , a journalist, a scientist , a politician , a health influencer, a content generator )
- incentive for the author
- reputation building
- audience building
- sell a product
- zero incentive (e.g. retired food researcher)
Do any of these figure in your analysis ? Could you recommendations any framework for critical thinking in our current information overloaded age?
That's a great question. I think the answer is complex, because I don't think there is a simple test we can use to distinguish reliable information from unreliable information. I'm always skeptical of "health hacks" to the extent that the presented as some easy but somehow secret solution to our problems. They're like get-rich-quick schemes. Journalists and scientists can be good sources, but you always want to check that what they're saying is in line with the scientific consensus and not just their iconoclastic view. In general credible health information is backed by highly cited peer reviewed articles (I always check for recent well-cited articles on Google Scholar). The consensus isn't always right, but any source who tells you to trust them over the consensus—or who implies there's some kind of a conspiracy to keep the truth from you—probably shouldn't be trusted. That's my view, in any case.
A key thing worth mentioning is the ample evidence that some people are much better at forecasting than others. And I understand it, these "superforecasters" are not typically relying on ultra-fancy mathematical models with minimal assumptions. They use vibes, numbers, and some arithmetic here and there. (The difference is that their vibes are of better quality than most people's.)
That's right. Research shows that there are "superforecasters" who are significantly and consistently more accurate than other forecasters (I was one of the forecasters identified as a "superforecaster" in the original Good Judgment Project tournament). While these skilled forecasters often use models to make forecasts, they don't rely on any single method outside of using their judgment about what they vibes indicate. As you say, the difference is that their judgment about the vibes is generally good.
Yeah I don't know how to have cut through with this concept but I think about it a lot. Vibeness seem orthogonal to how true something is. Some vibes are true, some robust models are false.
I guess I'd say that pretty much all models are false—in the sense that the simplify away features of reality—but some are useful. I think the same can be said of some interpretations of the vibes. In each case the real question is, do they yield accurate predictions?
Very intertesting thoughts.
Whether I am reading reviews to purchase a software or the next "health hack" in news media or election forcaster (like nate silver), I always puzzle about how to trust a piece of information.
Do you use any framework or heuristic for assessing a piece of information ?
For example, lets take a health hack, Resistant starch consumption as a means for glucose regulation. How do you frame/assess following parts of the context
- context of information ( social media feed, news article in mainstream media, post on less mainstream media , group discussion with actual humans, group discussion with virtual avatars , apnews , medical journal )
- message itself ( resistant starch is better for your health )
- benefits and cost of accepting message (this is personal i get it)
- platform ( ad somewhere, social media feed, mass media ) and their incentives
- author ( a reporter ? , a journalist, a scientist , a politician , a health influencer, a content generator )
- incentive for the author
- reputation building
- audience building
- sell a product
- zero incentive (e.g. retired food researcher)
Do any of these figure in your analysis ? Could you recommendations any framework for critical thinking in our current information overloaded age?
That's a great question. I think the answer is complex, because I don't think there is a simple test we can use to distinguish reliable information from unreliable information. I'm always skeptical of "health hacks" to the extent that the presented as some easy but somehow secret solution to our problems. They're like get-rich-quick schemes. Journalists and scientists can be good sources, but you always want to check that what they're saying is in line with the scientific consensus and not just their iconoclastic view. In general credible health information is backed by highly cited peer reviewed articles (I always check for recent well-cited articles on Google Scholar). The consensus isn't always right, but any source who tells you to trust them over the consensus—or who implies there's some kind of a conspiracy to keep the truth from you—probably shouldn't be trusted. That's my view, in any case.