The bot is also "experiencing severe hallucinations," a phenomenon in which AI confidently spits out inaccuracies like they're facts, the employees said.In Q's case, it could deliver such bad legal advice to "potentially induce cardiac incidents in Legal," as one employee put it in a company Slack channel, according to Platformer....It's not uncommon for generative AI chatbots to falter.It wasn't long after Microsoft released its consumer-focused generative AI assistant, Sydney, that it went viral with its own hallucinations. But Q's transgressions are all the more ironic given that the bot was designed to be a safer and more secure option that businesses could rely on.
The reason I think this problem is insurmountable is the same reason that Gödel's incompleteness theorems are true. Formal systems, including all algorithms, are closed and unable to prove their own completeness or coherence. All of these AI language model systems are systems of that type.
Unlike an ordinary mathematical or strict-logical system they have a vast number of assumptions: potentially up to and including everything ever written by anyone, anywhere, at any time. They might thus seem to be better able to reason about reality than any of us, because none of us have access to nearly that amount of data on which to model reality.
What we have that they lack, however, is a limited* ability to test the assumptions against reality. As we were discussing the other day, you always have to test the logical assumptions outside the sytem of logic, e.g. empirically. That is just what AI cannot do. They depend on human interaction to do that for them: it is for you to verify that the paper they just cited to you doesn't exist, and the author they claimed for it never lived. Even though these algorithms are deducting in a highly sophisticated manner, working off a vast set of assumptions built into a detailed model, they're still closed systems like formal logic or maths.
There are advances that are possible to give AI a limited but not-yet-extant method of checking itself; because I'm not in favor of the technology, I won't go into them. Even so, the best ones I can think of still depend on checking back against the model, and thus are incomplete. The ability to go outside the system is the thing they lack, and they will lack it unless they develop genuine consciousness.
That then might provide an answer to the question of the other day, about how you could tell if an AI was really conscious or not. Until they are, they'll hallucinate, and they won't be able to tell that they are.
*Cf. Kant's Critique of Pure Reason for an account of the limitations on this ability to check reality, the noumena, against the phenomena that we have in our minds. There is a chance that a higher order of beings could use that limit against us in the same way that we could use hallucination against AI, if we needed a weapon against it or a means of controlling it. You wouldn't know that they were doing so, if they existed, because you couldn't know.
3 comments:
Grim said-
"What we have that they lack, however, is a limited* ability to test the assumptions against reality. As we were discussing the other day, you always have to test the logical assumptions outside the system of logic, e.g. empirically."
Well, we just ran a three year test- humans failed miserably. Most lived the hallucination.
Formal systems, including all algorithms, are closed and unable to prove their own completeness or coherence.
Which is to say they cannot ask (nor answer) the question "Why?"
Well they can, but they can only answer it with data from inside the system.
What they really can't do is check outside. There is no outside for them. The best they can do is express their confident answer, and wait to see if we tell them from the outside that they're barking mad.
Post a Comment