Metaculus also run a forecasting contest in Q3 2024 with 55 bots and a $30k prize pool, where the humans came clearly on top: https://www.metaculus.com/notebooks/28784/ It'd be great if this was also mentioned in the post! This will be repeated in the next quarters, so we'll see if the bots improve.
Thanks for posting this link! I hadn't seen these benchmarking results. It's striking that Metaculus finds aren't appropriately scope sensitive, which is more evidence of limitations in their ability to reason. I'll look forward to reading the Q4 update.
Hey Robert I love your stuff, however I have to forecast that this post will not age well. ;-)
In my humble opinion you are not giving enough credit to the current rate of change. If current models already come close they will beat the average human in no time and the superforecasters very soon as well.
Would be great to have a prediction market on that one. :D
Thanks! You may be right. It wouldn't surprise me if a breakthrough got us there. But I don't think current models come that close, and I think LLMs probably aren't on their own the right tool to get us there.
Metaculus also run a forecasting contest in Q3 2024 with 55 bots and a $30k prize pool, where the humans came clearly on top: https://www.metaculus.com/notebooks/28784/ It'd be great if this was also mentioned in the post! This will be repeated in the next quarters, so we'll see if the bots improve.
Thanks for posting this link! I hadn't seen these benchmarking results. It's striking that Metaculus finds aren't appropriately scope sensitive, which is more evidence of limitations in their ability to reason. I'll look forward to reading the Q4 update.
Hey Robert I love your stuff, however I have to forecast that this post will not age well. ;-)
In my humble opinion you are not giving enough credit to the current rate of change. If current models already come close they will beat the average human in no time and the superforecasters very soon as well.
Would be great to have a prediction market on that one. :D
Thanks! You may be right. It wouldn't surprise me if a breakthrough got us there. But I don't think current models come that close, and I think LLMs probably aren't on their own the right tool to get us there.
I’m still waiting for SubStack to let me manage my accounts in the application. SubStack basically makes me insane.
What's the issue you're having with the Substack app? I really just use it to read. Do you have to use a browser to change account settings?
I don’t know. I think maybe I should just give up on it for now. But I HAVE to read Jeff Tiedrich because nobody trolls Trump better.