I not only think this isn’t literally true, but that it has already plateaued. One example being the disappointment that ChatGPT 5 was upon release. We have reached (and arguably passed) the limits of what can be practically done at scale. That is why people are so universally, apolitically and virulently opposed to data centers – they sense this without necessarily understanding it. It’s also why they place such a low value on LLM output: they are happy to use it when it’s free or trivially cheap, but ask them to actually contribute to the gigantic bottom line of LLM infrastructure, and they universally say, “nah”. Even developers, for the most part.
There is nothing in the technology that, e.g., AGI is going to just fall out of. Far smarter people than I have already unambiguously said that. What we are doing is throwing massive enough compute resources at the technology that it superficially resembles in many ways the skim-the-surface, unself-aware kind of thinking modes that many of us humans routinely use (and that often suffices). And as recent research has shown, it does not so much hallucinate as ingratiate. It subtly leads you to the option that it figures is least-uncomfortable to you, even when it is in adversarial mode.
In examining the reasoning chain of my local Qwen model, it will use the kinds of words humans would use to describe its deliberations and so it seems almost human (e.g. I accidentally started typing into what I thought was a MacOS terminal window when it was an external terminal window for the model, and it sounded positively frustrated that I wasn’t engaging with it, e.g., “I already told the user twice that … but I should tone down my response”).
We should no more be misled by this than people should consider an LLM to be a love interest or a therapist in anything like the true sense. If it helps someone to name and gender their chatbot, there’s nothing wrong with it inherently, but don’t think that it actually has feelings, or more importantly, understanding.
If we ever do break through that barrier, the philosophical side of me can only think that an important prerequisite to true consciousness is suffering. We will have the moral conundrum of creating entities that we will cause to suffer in various ways on the path to sentience. Doing that will likely involve some form of fairly sophisticated embodied consciousness that is currently beyond our reach, or else it will produce a form of consciousness that will be so alien – so inhuman – we won’t recognize it as such, nor find it that useful. Such a consciousness will still not really “get” our questions. Even if there are no dangerous aspects to that, it’s a barrier to unbounded improvement.