Humanity is a risky AGI goal

Today I'm talking about the discourse I see around LLM releases in places like Youtube, Hacker News and Reddit. Every time a new model comes out there is a rush to figure out the ways in which the models fails to do something that we would consider baseline for a functional human.

We wouldn't expect a human to write an reasonably coherent thesis on a topic based on large chunks of available information on the internet in half an hour, nor would we expect to be able to create an image that passes on first glance as real in a matter of seconds. By the same token it seems like expecting a language model to intuit (with no way to test in the real world) something like which side of the door an object is on is a bar that would require a mind that is in a fundamental sense super-human.

The way these models 'think' is inherently quite alien. Pushing on and improving them to achieve human like results across the board runs the risk of missing the ways in which they streak ahead and eclipse our ability to match or even understand what they are doing, and therein lies the risk I'm seeing.

We aren't there quite there yet, outside of games most if not all of what models have achieved so far could theoretically be done with enough human time and resources (even if an unrealistic amount) as far as I know, even in the case of novel math or biomedical solutions.

That doesn't mean it won't happen though, and once a frontier model is 'there' I would expect that it would be able to achieve a large number of results like this as easily as a current model can produce an uncanny Ghibli image or juxtapose Eminem and Shakespeare. If I've noticed anything about the current rate of improvement it's that some goal posts haven't moved, while others move almost as fast as we can create them (what happened to the Turing test).

Whether or not this is bad is in many ways besides the point, assuming this will happen regardless it seems more important to have ways to notice it sooner rather than later, and come up with ways to understand and extrapolate the nature, benefits, and risks of these superhuman 'abilities' as and when they come up.