The Turing Test has been the predominant means of judging the merits and capabilities of artificial intelligence. It suggests that if you can determine whether you’re conversing with an artificial intelligence, then you’re not conversing with anything truly intelligent.
Would a more accurate test be to test whether an AI can learn as effectively as a young child? The Allen Institute have developed a new app, called Aristo, which is being put through its paces in learning for the New York State’s standard science exams.
A learning intelligence
“We can put our understanding of progress in AI and in natural language on an objective footing,” the developers say. Being able to compare the merits of different approaches should make it easier to identify promising approaches and could accelerate progress, he says.
The Institute plan to launch a contest this month to build software that can successfully answer grade eight science questions, with a bounty of $50,000 offered to successful entrants.
The challenge is a significant one, as the existing software struggles with fourth grade questions, with a particular difficulty with any question involving images.
This lack of apparent common sense is something that holds back many AI systems, including the personal assistants built into most modern smartphones. All of them rely instead on rule based intelligence to construct what they believe to be the best answer to our queries.
The challenge is useful therefore as it allows AI researchers to benchmark where current understanding in the field is.
“What’s difficult for humans is very different from what’s difficult for machines,” the team say. “Standardized tests for humans don’t get very good coverage of the kinds of problems that are hard for computers.”
An exam for machines
A possible approach would be to create exam questions specifically for machines. An example of just such a question could be:
“Sally’s favorite cow died yesterday. The cow will probably be alive again a) tomorrow; b) within a week; c) within a year; d) within a few years; e) the cow will never be alive again.”
Of course, the search for common sense could be implied as existing within most school exams. Whether it’s mimicking existing tests or crafting custom exams however, it’s sure to be an interesting exploration into the next generation of AI.
I think at times we under-estimate just how much kids do know, so to get an AI up to that level would be seriously impressive.
The result may be impressive, but you can't say anything about the machine's "IQ". IQ tests are calibrated on humans (who are raised in a certain culture) and they naturally won't generalize to an AI.
For example, there are IQ tests where one simply repeats a sequence of characters backwards. For humans, this turns out to be pretty well correlated with general academic ability, but obviously it wouldn't work for an AI.
It'll be a sad day when he finally achieves an IQ test score of a young man and his parents have to tell him that he's not even a real boy 😉
It's not really an AI is it, it's just a predictable algorithm.