> defining AGI as matching the cognitive versatility and proficiency of a well-educated adult
I don't think people really realize how extraordinary accomplishment it would be to have an artificial system matching the cognitive versatility and proficiency of an uneducated child, much less a well-educated adult. Hell, AI matching the intelligence of some nonhuman animals would be an epoch-defining accomplishment.
After reading the paper I’m struck by the lack of any discussion of awareness. Cognition requires at its basis awareness, which due to its entirely non verbal and unconstructed basis, is profoundly difficult to describe, measure, quantify, or label. This makes it to my mind impossible to train a model to be aware, let alone for humans to concretely describe it or evaluate it. Philosophy, especially Buddhism, has tried for thousands of years and psychology has all but abandoned attempting so. Hence papers like this that define AGI on psychometric dimensions that have the advantage of being easily measured but the disadvantage of being incomplete. My father is an emeritus professor of psychometrics and he agrees this is the biggest hurdle to AGI - that our ability to measure the dimensions of intelligence is woefully insufficient to the task of replicating intelligence. We scratch the surface and his opinion is language is sufficient to capture the knowledge of man, but not the spark of awareness required to be intelligent.
This isn’t meant to be a mystical statement that it’s magic that makes humans intelligent or some exotic process impossible to compute. But that the nature of our mind is not observable in its entirety to us sufficient that the current learned reinforcement techniques can’t achieve it.
Try this exercise. Do not think and let your mind clear. Ideas will surface. By what process did they surface? Or clear your mind entirely then try to perform some complex task. You will be able to. How did you do this without thought? We’ve all had sudden insights without deliberation or thought. Where did these come from? By what process did you arrive at them? Most of the things we do or think are not deliberative and definitely not structured with language. This process is unobservable and not measurable, and the only way we have to do so is through imperfect verbalizations that hint out some vague outline of a subconscious mind. But without being able to train a model on that subconscious process, one that can’t be expressed in language with any meaningful sufficiency, how will language models demonstrate it? Their very nature of autoregressive inference prohibits such a process from emerging at any scale. We might very well be able to fake it to an extent that it fools us, but awareness isn’t there - and I’d assert that awareness is all you need.
The problem, I guess, with these methods is, they consider human intelligence as something detached from human biology. I think this is incorrect. Everything that goes in the human mind is firmly rooted in the biological state of that human, and the biological cycles that evolved through millennia.
Things like chess-playing skill of a machine could be bench-marked against that of a human, but the abstract feelings that drive reasoning and correlations inside a human mind are more biological than logical.
There’s already a vague definition that AGI is an AI with all the cognitive capabilities of a human. Yes, it’s vague - people differ.
This paper promises to fix "the lack of a concrete definition for Artificial General Intelligence", yet it still relies on the vague notion of a "well-educated adult". That’s especially peculiar, since in many fields AI is already beyond the level of an adult.
You might say this is about "jaggedness", because AI clearly lacks quite a few skills:
> Application of this framework reveals a highly “jagged” cognitive profile in contemporary models.
But all intelligence, of any sort, is "jagged" when measured against a different set of problems or environments.
So, if that’s the case, this isn’t really a framework for AGI; it’s a framework for measuring AI along a particular set of dimensions. A more honest title might be: "A Framework for Measuring the Jaggedness of AI Against the Cattell–Horn–Carroll Theory". It wouldn't be nearly as sexy, though.
The fundamental premise of this paper seems flawed -- take a measure specifically designed for the nuances of how human performance on a benchmark correlates with intelligence in the real world, and then pretend as if it makes sense to judge a machine's intelligence on that same basis, when machines do best on these kinds of benchmarks in a way that falls apart when it comes to the messiness of the real world.
This paper, for example, uses the 'dual N-back test' as part of its evaluation. In humans this relates to variation in our ability to use working memory, which in humans relates to 'g'; but it seems pretty meaningless when applied to transformers -- because the task itself has nothing intrinsically to do with intelligence, and of course 'dual N-back' should be easy for transformers -- they should have complete recall over their large context window.
Human intelligence tests are designed to measure variation in human intelligence -- it's silly to take those same isolated benchmarks and pretend they mean the same thing when applied to machines. Obviously a machine doing well on an IQ test doesn't mean that it will be able to do what a high IQ person could do in the messy real world; it's a benchmark, and it's only a meaningful benchmark because in humans IQ measures are designed to correlate with long-term outcomes and abilities.
That is, in humans, performance on these isolated benchmarks is correlated with our ability to exist in the messy real-world, but for AI, that correlation doesn't exist -- because the tests weren't designed to measure 'intelligence' per se, but human intelligence in the context of human lives.
Don't get me wrong, I am super excited about what AI is doing for technology. But this endless conversation about "what is AGI" is so boring.
It makes me think of every single public discussion that's ever been had about quantum, where you can't start the conversation unless you go through a quick 101 on what a qubit is.
As with any technology, there's not really a destination. There is only the process of improvement. The only real definitive point is when a technology becomes obsolete, though it is still kept alive through a celebration of its nostalgia.
AI will continue to improve. More workflows will become automated. And from our perception, no matter what the rapidness of advancement is, we're still frogs in water.
We have SAGI: Stupid Artificial General Intelligence. It's actually quite general, but works differently. In some areas it can be better or faster than a human, and in others it's more stupid.
Just like an airplane doesn't work exactly like a bird, but both can fly.
Is it about jobs/tasks, or cognitive capabilities? The majority of the AI-valley seems to focus on the former, TFA focuses on the latter.
Can it do tasks, or jobs? Jobs are bundles of tasks. AI might be able to do 90% of tasks for a given job, but not the whole job.
If tasks, what counts as a task: Is it only specific things with clear success criteria? That's easier.
Is scaffolding allowed: Does it need to be able to do the tasks/jobs without scaffolding and human-written few-shot prompts?
Today's tasks/jobs only, or does it include future ones too? As tasks and jobs get automated, jobs evolve and get re-defined. So, being able to do the future jobs too is much harder.
Remote only, or in-person too: In-person too is a much higher bar.
What threshold of tasks/jobs: "most" is apparently typically understood to mean 80-95% (Mira Ariel). Automating 80% of tasks is different to 90% and 95% and 99%. diminishing returns. And how are the tasks counted - by frequency, by dollar-weighted, by unique count of tasks?
Only economically valuable tasks/jobs, or does it include anything a human can do?
A high-order bit on many people's AGI timelines is which definition of AGI they're using, so clarifying the definition is nice.
1) defining intelligence is very difficult, almost impossible. Much more the artificial one
2) there are many types of human intelligence. Verbal is one of them and the closest to comparing with LLMs
3) machines (not only LLMs but all, like robots) excel where humans are bad and vice versa due to their different background, without exception. Comparing the two is totally meaningless and unfair for both. Let's have both complement the other.
4) AGI remains a valid target but we are still very far from it, like in other ones, as control the DNA and treat arbitrary genetic diseases, solve the earth resource problem and harness other planets, create a near perfect sociopolitical system with no inequality. Another Singularity is added in the list
5) I am impressed by how far a PC cluster has come up through "shuffling tokens" but on the other side I am pessimistic of how further it can reach having finate input/training data.
I don't think it's really AGI until you can simply task it with creating a new better version of itself and it can succeed in doing that all on its own.
A team of humans can and will make a GPT-6. Can a team of GPT-5 agents make GPT-6 all on its own if you give it the resources necessary to do so?
Isn't part of the cognitive versatility of a human how fast and well they can learn a new subject without having to ingest so much training content on it?
Like in order for an LLM to come close to a human proficiency on a topic, the LLM seems to have to ingest a LOT more content than a human.
That 10-axis radial graph is very interesting. Do others besides this author agree with that representation?
The weak points are speed and long-term memory. Those are usually fixable in computing system. Weak long-term memory indicates that, somehow, a database needs to be bolted on. I've seen at least one system, for driving NPCs, where, after something interesting has happened, the system is asked to summarize what it learned from that session. That's stored somewhere outside the LLM and fed back in as a prompt when needed.
None of this addresses unstructured physical manipulation, which is still a huge hangup for robotics.
I think if you can put an AI in a humanoid robot (control for appearance), and it can convince me that it's a human after interacting it for a couple of months (control for edgecases), I'd consider it AGI. Surely it might be "smarter than" a human, but for the purpose of my assessing whether it's AGI, interacting with something "way smarter" would be distracting and hamper the assessment, so it has to be "play human" for the purpose of the task. If it can do that, AGI, I'd say. That would be pretty cool. Surely, this is coming, soon.
It's easy: we have reached AGI when there are zero jobs left. Or at least non manual labor jobs. If there is a single non-physical job left, then that means that person must be doing something that AI can't, so by definition, it's not AGI.
I think it'll be a steep sigmoid function. For a long time it'll be a productivity booster, but not enough "common sense" to replace people. We'll all laugh about how silly it was to worry about AI taking our jobs. Then some AI model will finally get over that last hump, maybe 10 or 20 years from now (or 1000, or 2}, and it will be only a couple months before everything collapses.
What if the Wright brothers had to pass a “bird exam”? That’s how we’re defining AGI today. Stop grading feathers; start designing thrust. Check out my new post: "If This Is How We Define AGI, I'm Out" - https://open.substack.com/pub/paulochen/p/if-this-is-how-we-...
How about AFI - artificial fast idiot. Dumber than a baby, but faster than an adult. Or AHI - artificial human imitator.
This is bad definition, because human baby is already AGI when it's born and it's brain is empty. AGI is the blank slate and ability to learn anything.
I'm also frustrated by the lack of clear definitions related to AI.
Do you know what's more frustrating, though? Focusing so heavily on definitions that we miss the practicality of it (and I'm guilt of this at times too).
We can debate definitions of AGI, but given that we don't know what a new model or system is capable of until its built and tested in the real world we have more serious questions in my opinion.
Debates over AI risk, safety, and alignment are still pretty uncommon and it seems most are happy enough to accept Jevons Paradox. Are we really going to unleash whatever we do build just to find out after the fact whether or not its AGI?
I was going to make a mildly snide remark about how once it can consistently make better decision than average person, it is automatically qualifies, but the paper itself is surprisingly thoughtful in describing both: where we are and where it would need to be.
I always find it interesting how the majority of comments on threads like this on HN are dismissive of current AI systems as "gimmicks", yet some of the most successful people on the planet think it's worth plowing a trillion dollars into them.
I don't know who's right, but the dichotomy is interesting.
The "computer" on star trek TNG was basically agentic LLMs (it knows what you mean when you ask it things, and it could solve things and modify programs by telling it what changes to make)
Data on ST:TNG was more like AGI. It had dreams, argued for itself as a sentient being, created art, controlled its own destiny through decision making.
I think "our" mistake is that we wanted to make a modern human first, while being unable to make an animal or even a caveman, and we lost something in the leap-frog.
But we effectively have a database of knowledge that has become interactive thanks to reinforcement learning, which is really useful!
To define AGI, we'd first have to define GI. Humans are very different. As park rangers like to say, there is an overlap between the smartest bears and the dumbest humans, which is why sometimes people can't open bear-proof trash cans.
It's a similar debate with self driving cars. They already drive better than most people in most situations (some humans crash and can't drive in the snow either for example).
Ultimately, defining AGI seems like a fools errand. At some point the AI will be good enough to do the tasks that some humans do (it already is!). That's all that really matters here.
Filling forms is a terribly artificial activity in essence. They are also very culturally biased, but that fits well with the material the NNs have been trained with.
So, surely those IQ-related tests might be acceptable rating tools for machines and they might get higher scores than anyone at some point.
Anyway, is the objective of this kind of research to actually measure the progress of buzzwords, or amplify them?
This is fine for a definition of AGI, but it's incomplete. It misses so many parts of the cognition that make humans flexible and successful. For example, emotions, feelings, varied pattern recognition, propreception, embodied awareness, social skills, and navigating ambiguous situation w/o algorithms. If the described 10 spectrums of intelligence were maxed by an LLM, it would still fall short.
> defining AGI as matching the cognitive versatility and proficiency of a well-educated adult
Seems most of the people one would encounter out in the world might not posses AGI, how are we supposed to be able to train our electrified rocks to have AGI if this is the case?
If no one has created a online quiz called "Are you smarter than AGI?" yet based on the proposed "ten core cognitive domains", I'd be disappointed.
I wish them luck. Any consensus at all, on any definition at all, would be a boon to mankind. Unfortunately I am certain that all we have to look forward to is endless goal post shifting.
All models are wrong, but some are useful. However when it comes to cognition and intelligence we seem to be in the “wrong and useless” era or maybe even “wrong and harmful” (history seems to suggest this as a necessary milestone…anyone remember “humorism”?)
Maybe we need a new term. I mean AGI just means artificial general intelligence as opposed to specialised AI like chess computers and never came with a particular level it had to be. Most people think of it as human level intelligence so perhaps we should call it that?
You need some expertise in a field to see past the amazing imitation capabilities of LLMs and get a realistic idea of how mediocre they are. The more you work with it the less you trust it. This is not _it_.
Since everyone's spitballing their idea of AGI, my personal take is that AGI should be a fully autonomous system that have a stable self-image of some sort, can act on its own volition, understand the outcome of its actions, learn from cause-and-effect, and can continue doing so indefinitely.
So far, LLMs aren't even remotely close to this, as they only do what they are told to do (directly or otherwise), they can't learn without a costly offline retraining process, they do not care in the slightest what they're tasked with doing or why, and they do not have anything approximating a sense of self beyond what they're told to be.
Everyone has a definition and so have I. I would call it an AGI when i replace my smartphone and laptop with it. When my screen time is zero? Can AGI replace screens? Go figure.
Creative problem solving and commonsense physics are missing, among others.
It is a valuable contribution but the CHC theory from psychology that this is based on is itself incomplete.
By commonsense physics, I mean something like simulating interactions of living and non-living entities in 3D over time. Seems more complicated than the examples in the web site and in most tests used in psychometrics.
Creative problem solving with cognitive leaps required for truly novel research & invention could lie outside the rubrics as well. The criteria in CHC are essential but incomplete I believe.
Interesting read. I agree completely with their Introduction, that the definition of AGI is constantly shifting, and this leads to endless (and useless) debates.
What I find cool about the paper is that they have gathered folks from lots of places (berkley, stanford, mit, etc). And no big4 labs. That's good imo.
tl;dr; Their definition: "AGI is an AI that can match or exceed the cognitive versatility and proficiency of a well-educated adult."
Cool. It's a definition. I doubt it will be agreed on by everyone, and I can see endless debates about just about every word in that definition. That's not gonna change. At least it's a starting point.
What I find interesting is that they specifically say it's not a benchmark, or a test set. It's a framework where they detail what should be tested, and how (with examples). They do have a "catchy" table with gpt4 vs gpt5, that I bet will be covered by every mainstream/blog/forum/etc out there -> gpt5 is at ~50% AGI. Big title. You won't believe where it was one year ago. Number 7 will shock you. And all that jazz.
Anyway, I don't think people will stop debating about AGI. And I doubt this methodology will be agreed on by everyone. At the end of the day both extremes are more ideological in nature than pragmatic. Both end want/need their view to be correct.
I enjoyed reading it. Don't think it will settle anything. And, as someone posted below, when the first model will hit 100% on their framework, we'll find new frameworks to debate about, just like we did with the turing test :)
I find the nature of AGI discussion to be so narrow and tedious. Intelligence is incomprehensibly more than being able to generate text that looks convincingly like a human wrote it. The coordination of a physical body, the formation of novel thoughts, the translation of thoughts to action, understanding the consequences of those actions, and so on. There’s so much missing that is required to even approach a literal human infant’s “intelligence” that it feels like I’m going crazy entertaining people’s arguments that we are approaching “AGI”.
Ugh just looking at their list, this paper gets a hard no from me. Intelligence isn't mastery of some arbitrary list of mathematical subjects. It's the ability to learn and apply these subjects (or anything else) after minimal exposure to the topic.
For a bar as high as AGI (and not just 'the skills of an educated person,' which is what this paper seems to be describing), we should include abstract mathematical reasoning, and the ability to generate new ideas or even whole subfields to solve open problems.
I think a lot of this is all backwards. People think AGI is taking something dumb, like an LLM, and sticking on learning, like a software upgrade.
I think it's the other way around: you build a system that first and foremost _learns_ as part of its fundamental function, _then_ you train it in the domain you want expertise.
You're not going to get expertise in all domains all the time, just like with people. And you're not going to get a perfect slave either, just like with humans. You'll probably get something more like in between a human and machine. If that's what you really want, great.
To put this another way, if you neglect your kids, they're still going to learn things, just probably not things you want them to learn. If you neglect your language model it's just not going to do anything.
> Last, we deliberately focus on core cognitive capabilities
rather than physical abilities such as motor skills or tactile sensing, as we seek to measure the
capabilities of the mind rather than the quality of its actuators or sensors.
seems pretty unfair to exclude motor skills, especially given
1) how central they are to human economic activity, and
2) how moravec's paradox tells us they are the hard part.
A "general intelligence" is equivalent to a golden retriever or dolphin. A human general intelligence is a $3/hr minimum wage worker from some undeveloped country.
This is a serious paper by serious people and it is worth reading, but any definition of intelligence that depends on human beings as reference will never be a good basis for evaluating non human intelligence.
You could easily write the reverse of this paper that questions whether human beings have general intelligence by listing all the things that LLMs can do, which human beings can't -- for example producing a reasonably accurate summary of a paper in a few seconds or speaking hundreds of different languages with reasonable fluency.
You can always cherry pick stuff that humans are capable that LLMs are not capable of and vice versa, and and I don't think there is any reason to privilege certain capabilities over others.
I personally do not believe that "General Intelligence" exists as a quantifiable feature of reality, whether in humans or machines. It's phlogiston, it's the luminiferous ether. It's a dead metaphor.
I think what is more interesting is focusing on _specific capabilities_ that are lacking and how to solve each of them. I don't think it's at all _cheating_ to supplement LLM's with tool use, RAG, the ability to run python code. If intelligence can be said to exist at all, it is as part of a system, and even human intelligence is not entirely located in the brain, but is distributed throughout the body. Even a lot of what people generally think of as intelligence -- the ability to reason and solve logic and math problems typically requires people to _write stuff down_ -- ie, use external tools and work through a process mechanically.
> To operationalize this, we ground our methodology in Cattell-Horn-Carroll theory, the most empirically validated model of human cognition
Cattell-Horn-Carroll theory, like a lot of psychometric research, is based on collecting a lot of data and running factor analysis (or similar) to look for axes that seem orthogonal.
It's not clear that the axes are necessary or sufficient to define intelligence, especially if the goal is to define intelligence that applies to non-humans.
For example reading and writing ability and visual processing imply the organism has light sensors, which it may not. Do all intelligent beings have vision? I don't see an obvious reason why they would.
Whatever definition you use for AGI probably shouldn't depend heavily on having analyzed human-specific data for the same reason that your definition of what counts as music shouldn't depend entirely on inferences from a single genre.
A definition of AGI
(arxiv.org)301 points by pegasus 26 October 2025 | 502 comments
Comments
I don't think people really realize how extraordinary accomplishment it would be to have an artificial system matching the cognitive versatility and proficiency of an uneducated child, much less a well-educated adult. Hell, AI matching the intelligence of some nonhuman animals would be an epoch-defining accomplishment.
This isn’t meant to be a mystical statement that it’s magic that makes humans intelligent or some exotic process impossible to compute. But that the nature of our mind is not observable in its entirety to us sufficient that the current learned reinforcement techniques can’t achieve it.
Try this exercise. Do not think and let your mind clear. Ideas will surface. By what process did they surface? Or clear your mind entirely then try to perform some complex task. You will be able to. How did you do this without thought? We’ve all had sudden insights without deliberation or thought. Where did these come from? By what process did you arrive at them? Most of the things we do or think are not deliberative and definitely not structured with language. This process is unobservable and not measurable, and the only way we have to do so is through imperfect verbalizations that hint out some vague outline of a subconscious mind. But without being able to train a model on that subconscious process, one that can’t be expressed in language with any meaningful sufficiency, how will language models demonstrate it? Their very nature of autoregressive inference prohibits such a process from emerging at any scale. We might very well be able to fake it to an extent that it fools us, but awareness isn’t there - and I’d assert that awareness is all you need.
Things like chess-playing skill of a machine could be bench-marked against that of a human, but the abstract feelings that drive reasoning and correlations inside a human mind are more biological than logical.
This paper promises to fix "the lack of a concrete definition for Artificial General Intelligence", yet it still relies on the vague notion of a "well-educated adult". That’s especially peculiar, since in many fields AI is already beyond the level of an adult.
You might say this is about "jaggedness", because AI clearly lacks quite a few skills:
> Application of this framework reveals a highly “jagged” cognitive profile in contemporary models.
But all intelligence, of any sort, is "jagged" when measured against a different set of problems or environments.
So, if that’s the case, this isn’t really a framework for AGI; it’s a framework for measuring AI along a particular set of dimensions. A more honest title might be: "A Framework for Measuring the Jaggedness of AI Against the Cattell–Horn–Carroll Theory". It wouldn't be nearly as sexy, though.
This paper, for example, uses the 'dual N-back test' as part of its evaluation. In humans this relates to variation in our ability to use working memory, which in humans relates to 'g'; but it seems pretty meaningless when applied to transformers -- because the task itself has nothing intrinsically to do with intelligence, and of course 'dual N-back' should be easy for transformers -- they should have complete recall over their large context window.
Human intelligence tests are designed to measure variation in human intelligence -- it's silly to take those same isolated benchmarks and pretend they mean the same thing when applied to machines. Obviously a machine doing well on an IQ test doesn't mean that it will be able to do what a high IQ person could do in the messy real world; it's a benchmark, and it's only a meaningful benchmark because in humans IQ measures are designed to correlate with long-term outcomes and abilities.
That is, in humans, performance on these isolated benchmarks is correlated with our ability to exist in the messy real-world, but for AI, that correlation doesn't exist -- because the tests weren't designed to measure 'intelligence' per se, but human intelligence in the context of human lives.
It makes me think of every single public discussion that's ever been had about quantum, where you can't start the conversation unless you go through a quick 101 on what a qubit is.
As with any technology, there's not really a destination. There is only the process of improvement. The only real definitive point is when a technology becomes obsolete, though it is still kept alive through a celebration of its nostalgia.
AI will continue to improve. More workflows will become automated. And from our perception, no matter what the rapidness of advancement is, we're still frogs in water.
Also, weird to see Gary Marcus and Yoshua Bengio on the same paper. Who really wrote this? Author lists are so performative now.
Just like an airplane doesn't work exactly like a bird, but both can fly.
https://x.com/m2saxon/status/1979349387391439198
Is it about jobs/tasks, or cognitive capabilities? The majority of the AI-valley seems to focus on the former, TFA focuses on the latter.
Can it do tasks, or jobs? Jobs are bundles of tasks. AI might be able to do 90% of tasks for a given job, but not the whole job.
If tasks, what counts as a task: Is it only specific things with clear success criteria? That's easier.
Is scaffolding allowed: Does it need to be able to do the tasks/jobs without scaffolding and human-written few-shot prompts?
Today's tasks/jobs only, or does it include future ones too? As tasks and jobs get automated, jobs evolve and get re-defined. So, being able to do the future jobs too is much harder.
Remote only, or in-person too: In-person too is a much higher bar.
What threshold of tasks/jobs: "most" is apparently typically understood to mean 80-95% (Mira Ariel). Automating 80% of tasks is different to 90% and 95% and 99%. diminishing returns. And how are the tasks counted - by frequency, by dollar-weighted, by unique count of tasks?
Only economically valuable tasks/jobs, or does it include anything a human can do?
A high-order bit on many people's AGI timelines is which definition of AGI they're using, so clarifying the definition is nice.
1) defining intelligence is very difficult, almost impossible. Much more the artificial one
2) there are many types of human intelligence. Verbal is one of them and the closest to comparing with LLMs
3) machines (not only LLMs but all, like robots) excel where humans are bad and vice versa due to their different background, without exception. Comparing the two is totally meaningless and unfair for both. Let's have both complement the other.
4) AGI remains a valid target but we are still very far from it, like in other ones, as control the DNA and treat arbitrary genetic diseases, solve the earth resource problem and harness other planets, create a near perfect sociopolitical system with no inequality. Another Singularity is added in the list
5) I am impressed by how far a PC cluster has come up through "shuffling tokens" but on the other side I am pessimistic of how further it can reach having finate input/training data.
A team of humans can and will make a GPT-6. Can a team of GPT-5 agents make GPT-6 all on its own if you give it the resources necessary to do so?
Like in order for an LLM to come close to a human proficiency on a topic, the LLM seems to have to ingest a LOT more content than a human.
That 10-axis radial graph is very interesting. Do others besides this author agree with that representation?
The weak points are speed and long-term memory. Those are usually fixable in computing system. Weak long-term memory indicates that, somehow, a database needs to be bolted on. I've seen at least one system, for driving NPCs, where, after something interesting has happened, the system is asked to summarize what it learned from that session. That's stored somewhere outside the LLM and fed back in as a prompt when needed.
None of this addresses unstructured physical manipulation, which is still a huge hangup for robotics.
I think it'll be a steep sigmoid function. For a long time it'll be a productivity booster, but not enough "common sense" to replace people. We'll all laugh about how silly it was to worry about AI taking our jobs. Then some AI model will finally get over that last hump, maybe 10 or 20 years from now (or 1000, or 2}, and it will be only a couple months before everything collapses.
This is bad definition, because human baby is already AGI when it's born and it's brain is empty. AGI is the blank slate and ability to learn anything.
Do you know what's more frustrating, though? Focusing so heavily on definitions that we miss the practicality of it (and I'm guilt of this at times too).
We can debate definitions of AGI, but given that we don't know what a new model or system is capable of until its built and tested in the real world we have more serious questions in my opinion.
Debates over AI risk, safety, and alignment are still pretty uncommon and it seems most are happy enough to accept Jevons Paradox. Are we really going to unleash whatever we do build just to find out after the fact whether or not its AGI?
I don't know who's right, but the dichotomy is interesting.
The "computer" on star trek TNG was basically agentic LLMs (it knows what you mean when you ask it things, and it could solve things and modify programs by telling it what changes to make)
Data on ST:TNG was more like AGI. It had dreams, argued for itself as a sentient being, created art, controlled its own destiny through decision making.
It's a similar debate with self driving cars. They already drive better than most people in most situations (some humans crash and can't drive in the snow either for example).
Ultimately, defining AGI seems like a fools errand. At some point the AI will be good enough to do the tasks that some humans do (it already is!). That's all that really matters here.
You need context from internal system state that isn’t faked with a giant context window.
So, surely those IQ-related tests might be acceptable rating tools for machines and they might get higher scores than anyone at some point.
Anyway, is the objective of this kind of research to actually measure the progress of buzzwords, or amplify them?
an entity which is better than any human at any task.
Fight me!
Seems most of the people one would encounter out in the world might not posses AGI, how are we supposed to be able to train our electrified rocks to have AGI if this is the case?
If no one has created a online quiz called "Are you smarter than AGI?" yet based on the proposed "ten core cognitive domains", I'd be disappointed.
[1]: The capability to continually learn new information (associative, meaningful, and verbatim). (from the publication)
So far, LLMs aren't even remotely close to this, as they only do what they are told to do (directly or otherwise), they can't learn without a costly offline retraining process, they do not care in the slightest what they're tasked with doing or why, and they do not have anything approximating a sense of self beyond what they're told to be.
It is a valuable contribution but the CHC theory from psychology that this is based on is itself incomplete.
By commonsense physics, I mean something like simulating interactions of living and non-living entities in 3D over time. Seems more complicated than the examples in the web site and in most tests used in psychometrics.
Creative problem solving with cognitive leaps required for truly novel research & invention could lie outside the rubrics as well. The criteria in CHC are essential but incomplete I believe.
What I find cool about the paper is that they have gathered folks from lots of places (berkley, stanford, mit, etc). And no big4 labs. That's good imo.
tl;dr; Their definition: "AGI is an AI that can match or exceed the cognitive versatility and proficiency of a well-educated adult."
Cool. It's a definition. I doubt it will be agreed on by everyone, and I can see endless debates about just about every word in that definition. That's not gonna change. At least it's a starting point.
What I find interesting is that they specifically say it's not a benchmark, or a test set. It's a framework where they detail what should be tested, and how (with examples). They do have a "catchy" table with gpt4 vs gpt5, that I bet will be covered by every mainstream/blog/forum/etc out there -> gpt5 is at ~50% AGI. Big title. You won't believe where it was one year ago. Number 7 will shock you. And all that jazz.
Anyway, I don't think people will stop debating about AGI. And I doubt this methodology will be agreed on by everyone. At the end of the day both extremes are more ideological in nature than pragmatic. Both end want/need their view to be correct.
I enjoyed reading it. Don't think it will settle anything. And, as someone posted below, when the first model will hit 100% on their framework, we'll find new frameworks to debate about, just like we did with the turing test :)
For a bar as high as AGI (and not just 'the skills of an educated person,' which is what this paper seems to be describing), we should include abstract mathematical reasoning, and the ability to generate new ideas or even whole subfields to solve open problems.
I think it's the other way around: you build a system that first and foremost _learns_ as part of its fundamental function, _then_ you train it in the domain you want expertise.
You're not going to get expertise in all domains all the time, just like with people. And you're not going to get a perfect slave either, just like with humans. You'll probably get something more like in between a human and machine. If that's what you really want, great.
To put this another way, if you neglect your kids, they're still going to learn things, just probably not things you want them to learn. If you neglect your language model it's just not going to do anything.
Right. That explains it.
seems pretty unfair to exclude motor skills, especially given 1) how central they are to human economic activity, and 2) how moravec's paradox tells us they are the hard part.
AGI was already here the day ChatGPT released: That's Peter Norvig's take too: https://www.noemamag.com/artificial-general-intelligence-is-...
https://en.wikipedia.org/wiki/Cattell%E2%80%93Horn%E2%80%93C...
If a person has all those criteria, they are superintelligent. They are beyond genius.
The AGI definition problem is that everyone keeps conflating AGI with ASI, Artificial Super Intelligence.
You could easily write the reverse of this paper that questions whether human beings have general intelligence by listing all the things that LLMs can do, which human beings can't -- for example producing a reasonably accurate summary of a paper in a few seconds or speaking hundreds of different languages with reasonable fluency.
You can always cherry pick stuff that humans are capable that LLMs are not capable of and vice versa, and and I don't think there is any reason to privilege certain capabilities over others.
I personally do not believe that "General Intelligence" exists as a quantifiable feature of reality, whether in humans or machines. It's phlogiston, it's the luminiferous ether. It's a dead metaphor.
I think what is more interesting is focusing on _specific capabilities_ that are lacking and how to solve each of them. I don't think it's at all _cheating_ to supplement LLM's with tool use, RAG, the ability to run python code. If intelligence can be said to exist at all, it is as part of a system, and even human intelligence is not entirely located in the brain, but is distributed throughout the body. Even a lot of what people generally think of as intelligence -- the ability to reason and solve logic and math problems typically requires people to _write stuff down_ -- ie, use external tools and work through a process mechanically.
Cattell-Horn-Carroll theory, like a lot of psychometric research, is based on collecting a lot of data and running factor analysis (or similar) to look for axes that seem orthogonal.
It's not clear that the axes are necessary or sufficient to define intelligence, especially if the goal is to define intelligence that applies to non-humans.
For example reading and writing ability and visual processing imply the organism has light sensors, which it may not. Do all intelligent beings have vision? I don't see an obvious reason why they would.
Whatever definition you use for AGI probably shouldn't depend heavily on having analyzed human-specific data for the same reason that your definition of what counts as music shouldn't depend entirely on inferences from a single genre.