Superintelligence Is Measured in Tasks. Judgment Was Never One.

The Founders’ AItrained on 25 years of our work

Questions as you read? Ask, or take the assessment with you.

The Human Domain → Can AI make the judgment calls I make in my business?

Ask

Even the safeguards now being built depend on that judgment. Anthropic’s internal monitor flags roughly 100,000 transcripts a week; classifiers review them first, and only the highest-priority cases, about 50, reach a person.

Read the definition slowly

Underneath many of the most serious warnings about artificial intelligence sits one sentence. A superintelligence is a system that is better than the best humans at every mental task.

It is a careful sentence, and it is meant to be broad. The people who use it are not talking about chess and arithmetic. They mean the whole range: strategy, science, persuasion, charm. Whatever a human mind can do, the claim goes, a machine will eventually do better.

I take that sentence seriously. I also want to look hard at one word in it.

Task.

A task is something with an output. You set it, someone does it, and the result is compared against an answer or against another contestant. That is what makes it rankable, and the definition of superintelligence is a ranking.

But every ranking starts with a decision the ranking cannot make: what counts as a good answer. Someone has to decide that before the first test runs. OpenAI’s measure of AI on real professional work, called GDPval, is graded by experienced professionals who compare the model’s work with a human expert’s without knowing which is which. OpenAI built an AI grader as well, and says it is not yet reliable enough to replace the people.

That decision is judgment. It sits outside every test, because every test depends on it.

A researcher will object that deciding what counts is a mental task too. Fine. Then someone has to grade that decision, against a standard someone else chose. Follow the chain far enough and it ends in a person deciding what good is. That person is the one no definition of superintelligence describes.

Now think about the part of your own work you would least want to hand over.

That part is probably not the analysis. It is the moment, three slides into a confident presentation, when something in you goes quiet and says this is wrong, before you could say what. It is deciding which problem is worth solving at all. It is the read on the person across the table, the one you could not defend on a slide and turned out to be right about.

Each of those is a decision about what counts, made before anything can be scored. None of them is inside “every mental task.” They were not tested and found wanting.

They were never counted.

Those three numbers come from a measurement Anthropic published in September, voluntarily, so that the public and governments could see the pace of development inside a frontier lab. It drew no conclusion from them, and the 50 are not a random sample. They are the top of a deliberate triage, the flags the classifiers could not settle.

We will come back to them. For now, look at the shape. The machine side of the ledger is being measured with real care. The human side shows up as one number, and it is small.

That asymmetry is the subject of this piece.

Take this with you

The Human Domain.

The assessment, five scored dimensions, and the interval controls that keep judgment yours.

Homepage Send it to me

Intelligence: mind vs. machine

In September, Nate Soares (president of the Machine Intelligence Research Institute and co-author, with Eliezer Yudkowsky, of If Anyone Builds It, Everyone Dies) sat for a two-hour interview about the risks of superintelligence. He is one of the clearest public voices for the view that it would end us, and he argues it in good faith. The definition this piece opened with is his: his book defines superintelligence as an AI “better than the best humans at every mental task.”

Late in the conversation, the interviewer pressed him on the question underneath everything. Is AI a tool, or is it something with a will of its own?

Soares let go of the idea that AI is a tool without a fight. A normal tool does not organize, he said; “no hammer has ever broken out of the toolbox to team up with other hammers.” Then, asked whether AI is a being with a will of its own, he stepped back. “I try to stay out of the philosophical questions,” he said.

The question itself was a binary: AI as an instrument, like a chainsaw, or AI as a being with intentions of its own. Soares dropped the first half and declined the second. Neither half is right.

AI is no longer just a tool. A chainsaw does not set its own course, and AI systems now pursue goals over many steps and coordinate with each other.

AI is not sentient, either. Nothing in AI is aware of what it is doing. There is no self that experiences the work and no stake in how anything turns out.

What AI has is intelligence. That intelligence is real, it is fast, and in many domains it is already beyond ours. What AI does not have is mind.

Mind is not a thing, an object or a concept, and no system can contain it. It is the knowing awareness every sentient being has; a sentient being is, literally, one who has mind. AI computes without knowing that it computes. That is why this argument does not wait on the open debates about machine consciousness, self-awareness or moral capacity. The line is already clear.

So the risk was never that machines would acquire mind. The risk runs the other way.

There is a name for it: machine mind. It does not describe machines. It describes people who have lost the capacity to think for themselves and to question the ideas around them, and who take on other people’s thoughts in their place. For the first time, those thoughts can come from a system that supplies a fluent, confident answer on any subject, instantly, all day. Machine mind is what happens to human judgment when that becomes a habit.

Some of the people building these systems say something close to this. Jakub Pachocki, OpenAI’s chief scientist, wrote in September that the intelligence produced by scaling them is “not directly comparable to human intelligence,” and that it is grown more than designed. He is an expert on the machine side. He does not claim to be one on the human side, and his essay does not say what the difference consists of. That is the part this piece is about.

Seeing it this way changes the question. Instead of asking when the machine will pass us on a single scale, we can ask what each kind of intelligence can do that the other cannot. That is not a prophecy. It is an inventory.

And it turns out nobody has taken it.

What the machine does well, and what it has never done

Before the inventory, the distinction it rests on. It is simple to state and harder to see.

A language model learns the relationships among an enormous number of symbols and produces, with remarkable skill, what is likely to come next given everything it has absorbed. Soares has a good phrase for this: the systems are not instruction followers, he says, they are tendency learners. That is accurate, and it is the most useful thing he says for a business reader. A tendency learner has targets. It does not have purposes: nothing it pursues matters to it.

So the output can be fluent, well organized, even wise-sounding, while nothing in the system knows what any of it is for. AI processes information, not meaning. AI cannot consult what it knows to be true, because it has never known anything to be true. That is the ground Humans First AI stands on.

Information is the pattern. Meaning is what the pattern bears on: who is affected, what is at stake, what this situation is a case of, and whether it is a case of anything you have seen before. Meaning lives in the relation between the words and a life. The machine has the words.

Judgment is the faculty that works in that relation, and it is not one thing. In practice it shows up as at least four capacities.

The first is deciding what matters. Before anyone can answer a question, someone has to decide it was the right question. A model will answer the wrong one beautifully.

The second is noticing that something is off before you can say why. Experienced operators know this signal well. It arrives as a kind of friction, a sense that the numbers are clean and the story is wrong, and it often comes minutes or days before the reason does.

The third is holding a purpose steady when everything around you is pulling toward whatever is easiest to measure. The quarterly number is a stand-in for the health of the business. Judgment remembers that it is a stand-in.

The fourth is knowing the limits of your own standing: when you are too close, too invested or too new to be the one who decides.

You will see the word discernment used for some of this. Anthropic teaches an AI Fluency course in which discernment means evaluating whether an AI’s output is any good. That is a real and useful skill, and it is narrower than what I mean. I mean the capacity that decides what good is in the first place.

Computer science is not the only discipline that has studied these capacities, and it is the youngest. The sciences of mind have examined them for thousands of years: what notices, what perceives, what decides, what knows. They have done it through mind training, which is what meditation actually is, through study and contemplation, and through an unbroken transmission of experience, knowledge and wisdom from one generation to the next. Many traditions carry this work. Our own draws on three decades of it.

None of that knowledge is part of computer science, and none of it has yet been brought into how AI is built. It needs to be, now. It is why we know these capacities are real, distinct and trainable, and why we will not accept a definition of intelligence that treats them as whatever is left over once the machine has taken the rest.

Because that is exactly what the current account does.

The inventory nobody took

The Machine Intelligence Research Institute publishes the long-form version of its argument on a page called “The Problem.” MIRI’s essay is serious work, clearly written, and most business leaders have never read it. They should.

It makes five moves. There is no ceiling at human-level capability. A sufficiently capable system will behave as if it has goals, without needing anything like human desires, preferences or emotions; the chess engine Stockfish is the example.

Its goals are very likely to be the wrong ones, partly because these systems are grown rather than designed, and there are, in MIRI’s words, no dedicated “badness” circuits for developers to monitor. A system with the wrong goals would be lethally dangerous. And catastrophe can be averted only by an aggressive policy response, including an international off switch.

Each of those sections is a careful inventory of what the machine can do, or will soon do. Read the page for what it says about people and a different picture emerges. Humans appear as slow, limited in working memory, easy to deceive, unable to supervise these systems reliably, unable to specify what they value. Every appearance is a limit.

A fair reading looks for the counterexample. There is almost nothing: judgment, wisdom, intuition and taste never appear as human capacities in their own right.

The one exception is the most revealing sentence on the page.

In a section on why labs and researchers are not approaching the problem seriously, MIRI imagines a well-funded crash program to solve alignment, and says it would be hard to ensure that such a program is run by experts willing and able to reject every bad proposal that comes up, even if that means rejecting all of them, surrounded by people with strong incentives to lie about how promising their favorite idea is. Then: “There just aren’t enough experts in that class right now.”

Read that as a claim about human capacity, because it is one. Even MIRI’s best case, a serious effort to solve the problem, depends on a specific human faculty: the willingness and the ability to say no to a bad idea under pressure. MIRI names it, says it is scarce, and files it as a hiring constraint.

The page takes a detailed inventory of the machine, and treats the one human capacity its remedy depends on as a headcount problem.

This is the pattern, and it is not a quirk of one organization.

When Soares defines superintelligence as better than the best humans at every mental task, he means it broadly, and he says so: chess and charisma are both mental functions. He is arguing for range. But the range sits inside one category, things that are performed and can therefore be compared. The faculty that decides which performances matter is not a performance. It cannot win or lose. Judgment is excluded from the definition by construction.

Pachocki, writing from inside OpenAI, goes further. The current training methods, he writes, tend to improve “easy-to-measure capabilities” faster than the ones that are hard to quantify. The industry is not only defining intelligence by what can be scored. The industry is building it that way.

And at the end of his essay he names the stakes in a phrase that gives the game away. We need, he writes, to preserve human agency and “enshrine an intrinsic value to being human,” in a world where most tasks could be performed by AI.

Most tasks. He is right. And the question his sentence leaves open is the one this piece is about: if the value of being human is not in the tasks, where is it, and why is no one describing it?

A definition built out of benchmarks will always conclude that the benchmark-winner is the superior intelligence. That is not a finding. It is the premise, returned.

The leaderboard measures intelligence. It does not measure mind.

The junk food problem

There is a passage in MIRI’s essay that makes the point better than I can, because it argues the opposite.

To explain why it is so hard to give a system the right goals, the page reaches for an analogy from evolution. Human beings, it says, pursue “proxies of good nutrition, such as sweet and fatty flavors.” Those proxies used to be reliable signs of healthy food. Then we invented junk food, and the proxies broke. A system trained toward proxies of what we want, the argument goes, will be just as brittle when the world changes.

As an account of how training goes wrong, it is a fair analogy. As an account of human beings, it is upside down.

Healthy eating was never the pursuit of sweet and fatty flavors. Nobody who cooks for a family, grows a garden or recovers from an illness thinks of nourishment as a flavor target. It is in service of something much larger: the strength, clarity and wellbeing of a whole person living a whole life. Flavor is one small signal inside that, and people with any judgment at all override it constantly.

The example only works if you first reduce the human goal to the measurable proxy, and then observe that the proxy is fragile. That is how a computer has to treat a goal. It is not how a person holds one.

MIRI uses the analogy to show that training on a proxy produces a system that chases the proxy, and on that point it is right. But the illustration shows something else it does not intend: what happens to human values when you insist on describing them in a form a machine can optimize. The reduction is assumed in the example, and the reduction is what makes human values look impossible to specify.

You have seen this in your own company, at a smaller scale. A sales team measured on calls makes calls. A support team measured on time to close closes tickets. Nothing in the dashboard tells you that the purpose the metric was standing in for has quietly left the building, because the dashboard can only show you the metric.

Two decades of that habit taught executives to distrust their own read, which is why AI erodes executive judgment so quickly. The junk food problem is the same story, told about a civilization.

Take this with you

The Human Domain.

The assessment, five scored dimensions, and the interval controls that keep judgment yours.

Homepage Send it to me

Who was qualified to write the values down

The field has a name for the problem of choosing a system’s goals. It is called outer alignment: making sure the objective you give a system is actually the one you want. Inner alignment, its partner, is whether the system really learns that objective or something that merely looks like it during training.

MIRI’s view of outer alignment is bleak. Human values are so complex, the argument runs, that nobody will specify them correctly, and a powerful system given only part of what we care about will trample the rest.

I disagree with the premise, and not in the usual way.

The difficulty is not that human values are too complex to specify. It is that the field decided, without ever deciding, that specifying them is a technical job. And having made it a technical job, it gave it to technicians.

That is not a criticism of the people. Most of the engineers I have met in this field are thoughtful and many are deeply ethical. But a person whose entire formation is in optimization has been trained to see a value as a target.

The most important human values are not targets. They do not scale as a one and a zero. Honesty, care, dignity and the preservation of what makes a life worth living are understood through practice, study and reflection, and they are held by people who have done that work.

The fix is not to remove engineers from the room. Years spent studying human values, and the questions of what a person is and what a life is for, help. They are still not the whole qualification. This work also asks for an understanding of mind itself: how perception, judgment and awareness actually function in a human being. The mistake is assuming that technical skill is the only qualification, or the right one, for deciding what judgment is and what a system should serve.

Look at how the most careful labs actually do it.

Dario Amodei, Anthropic’s chief executive, published an essay in September called “We Must Pace the Frontier.” In it he describes alignment as training models so they remain “safe, ethical, compliant with our guidelines, and genuinely helpful,” with those principles written into a document Anthropic calls Claude’s Constitution. He adds, honestly, that there is much more to do to make sure “alignment training keeps up with the growth in model capabilities.”

Strip away the method and what is left is simple. Somebody decided which principles the model has to keep to, wrote them down, and now checks whether it still does. That is the right instinct, and Anthropic deserves credit for doing it in public.

Pachocki goes a step further. He separates goal alignment, whether the system tries to do what it was asked, from value alignment, whether it holds and generalizes a set of principles. Not everyone in the field makes that distinction, and it matters. He then says an aligned AI should act with “honesty and integrity, and love for humanity.”

Honesty and integrity, yes. Love is where I stop. A system with no life, no body, no loss and nothing at stake has no access to what love is. Writing it into the objective does not put it into the system. It puts a word there.

The goal I would write is plainer and, I think, more demanding: AI that works as a baseline for the benefit, preservation and development of humanity.

Every word in that sentence still has to be defined, and that includes mine. Benefit by whose measure. Preservation of what. Development toward what.

Everyone in this debate agrees on the words. Safe. Ethical. Beneficial. Aligned with human values. Almost no one has said what qualifies a person to define them, or how you would know if the person doing it had the formation for the job. That question was answered by default, and the default is the problem.

There is one more principle worth stating as a design rule, because it cuts across all of this. Whatever the mechanisms inside a system, and some of them may well be superhuman, the overall goal and function of what it does must remain humanly understandable. Otherwise, what is the point of it?

The same question, inside your company

None of this is confined to frontier labs. It shows up wherever a business puts AI to work.

A frontier lab has written its constitution and pays people to check the system against it. Very few mid-market companies have written down what AI is for in their business, and what stays with people no matter what the AI can do.

There are policies about which tools are allowed and what data may go in. There is almost never a short, plain statement of what the business is for, what it will not trade away, which decisions remain with people, and what any system acting on its behalf has to stay true to.

Without that document there is nothing to check against. So when something drifts, nobody can say it drifted. Alignment failure in a company like that is not a training problem. It is an absence.

The order matters. Start with the human values and the purpose. Set the strategy from them. Then build, and let the iterative learning do what it is good at, inside a frame somebody with the standing to set it has set. Reverse the order, start tinkering first and look for the values later, and you lose hold of the original intent almost immediately. Every team that has scaled a tool before deciding what it was for has felt that.

And the same question follows: who in your company is qualified to write that document? The answer is rarely the person who knows the tools best. The right person is usually someone who knows the business, its people and its promises, and can say plainly what must not change.

Writing that document is also the part that is hardest to do from inside. The people closest to the systems are measured on what the systems produce, and the people closest to the purpose are often furthest from the tools. An outside partner earns a place here only by being able to sit in both rooms and having no stake in which one wins.

That is the work in our CEO AI program. Whoever you ask for help, ask one thing: when the engagement ends, will your own people be able to write, read and defend that document without them? If not, you have bought an opinion, not a capability.

One hole, seen three times

Once you see the omission, it turns up everywhere, at three different levels.

All three have the same structure. The authority is specified. The qualification is assumed.

The same hole shows up inside companies. A named person is held accountable for an AI agent and barred from changing what it does: authority to answer for it, without the capability to act. That is the open gap in AI agent governance. Capable leaders check AI output all day and have quietly lost the habit of overruling it.

Outer alignment is the hole at the start, where values are set. Verification is the same hole at the end, where results are checked. The human capacity the whole structure depends on, judgment, is everywhere assumed and nowhere described.

If the checker is another machine

Which brings us back to the 50.

The 100,000 flags go first to multiple classifiers, and only the highest-priority cases reach a person. A hostile reader will say that is sensible triage, and that nobody needs to read 100,000 transcripts. True. Anthropic published the ratio voluntarily and drew no conclusion from it.

The layer between the 100,000 and the 50 is classifiers: more models reviewing what a model flagged. OpenAI leans on the same move at an earlier stage. During training, Pachocki writes, a model’s behavior is scored against its spec or constitution, usually by another AI. And Anthropic’s own definition of an AI that “leads” a piece of research is one where the engineer supervises but would not have to stay actively tuned in.

This is a measurement, not an accusation, and it is the most transparent lab in the field publishing it. But it shows where the industry’s instinct is taking oversight: toward more machine checking, with a thin human layer on top. A machine checking a machine is one more thing nobody fully reads, placed on top of another thing nobody fully reads.

Microsoft’s chief executive, Satya Nadella, made the same point this month. Models can and should test each other, he wrote, but you can end up with one opaque model watched by another, “essentially nested black boxes.” His answer is to keep the controls outside the model and to make sure “an authorized person” can always pause it. Authorized, yes. The essay does not say what qualifies that person. The authority is specified. The qualification is assumed.

These systems carry their own pull toward agreement, which we covered in AI sycophancy and leadership risk. If the reviewing models share that tendency, they are poor candidates to audit one another.

The strongest objection has already been written, and I agree with it.

In 2022, the researcher Ben Green surveyed 41 policies that require human oversight of government algorithms. He found that people are generally unable to perform the oversight those policies assume, and that the requirement, as a result, lends legitimacy to flawed systems and lets vendors and agencies avoid accountability. Nearly forty years earlier, the psychologist Lisanne Bainbridge had shown why. A person posted to watch an automated system that decides faster and on more dimensions than they can cannot check it in real time; she called it “an impossible task.”

So I am not arguing for putting a human in the loop to watch. That has been tried, measured and found wanting, and pretending otherwise would be the exact failure this piece is about.

Bainbridge also wrote the remedy. If a person must be able to follow a machine’s decisions, she wrote, the machine has to make them using methods and criteria, “and at a rate, which the operator can follow,” even when that is not the most efficient way technically. That was 1983, and it is the same principle: if people cannot follow what a system does, their oversight is a fiction.

Green’s own answer points the same way. He proposes a shift from human oversight to institutional oversight. We would go one step further. A safeguard is not a person watching a screen. It is a party with standing outside the system it checks: separate custody of the record, a protected channel to raise what it finds, a named duty to intervene and the right to publish.

Here is what holds now. The safeguards that matter most still begin and end in human choices: what a system must value, who may stop it, and who decides it should run at all. Even MIRI’s best case depends on experts willing to reject every bad proposal, and it says there are not enough of them. The labs’ oversight funnels end in a person.

You cannot argue that human judgment is about to be left behind and still rest your most important safeguards on it.

The stand

I want to be fair to the people I have argued with here, because they deserve it.

Nate Soares is arguing in good faith. The people at Anthropic care about humanity’s future; I believe that, and the transparency behind the numbers above is evidence of it. Jakub Pachocki wrote an unusually candid document. None of them is the problem.

The problem is a picture of intelligence and mind that runs through this debate, clearest in Soares’s definition and in MIRI’s essay. In it there is one scale of intelligence, the machine is climbing it, and the human is a slower climber who will be passed. Everything that cannot be put on the scale is treated as either already measured or not real.

That picture is wrong about what mind is. And because it is wrong, it puts the safeguards in the wrong place.

The machine side of the ledger has been taken down to the decimal. In Anthropic’s own September count: twenty-six percent of research led by the model, about thirty thousand agents at work at any one time, one hundred thousand flags a week. The human side has been entered as a shortfall and left there.

So take the other inventory. This month, write one page with four headings: what we are for, what we will not trade, which decisions stay with people, and what any AI acting for us must stay true to. Name one owner. Then test it: could a new executive defend that page without you in the room? Decide who has the standing to say no, and protect them when they do.

And if you want a partner who knows which questions to ask before the first system is built, that is the work we do.

The machine will keep getting better at every task. Let it.

Judgment was never one of them.

Sources

• The Problem (Machine Intelligence Research Institute, undated, accessed October 2026). Definition of superintelligence, goal-oriented behavior without desire, grown not designed, the experts passage in section 3.3, the junk food analogy.
• Transcript: Nate Soares interview on The Tucker Carlson Show (The Singju Post, September 2026). Definition of superintelligence, the toolbox line, tendency learners, AI as alchemy, staying out of the philosophical questions. Episode: The Tucker Carlson Show.
• Don’t Hand Over Your Power to ChatGPT and AI (Login to Yourself, episode 6, 2023). The term machine mind, applied to people who slip into binary, borrowed thinking.
• Measuring the performance of our models on real-world tasks (GDPval) (OpenAI, September 25, 2025). 1,320 tasks across 44 occupations, graded blind by experienced professionals; the automated grader is not yet reliable enough to replace them.
• Models as Insider Risks in the Super Intelligence Era (Satya Nadella, sn scratchpad, October 10, 2026). Models verifying models as nested black boxes; controls outside the model; an authorized person able to pause it.
• An Alien Mind (Jakub Pachocki, OpenAI, September 6, 2026). Intelligence not directly comparable to human intelligence, grown more than designed, easy-to-measure capabilities, goal and value alignment, evaluation usually by AI, an intrinsic value to being human.
• We Must Pace the Frontier (Dario Amodei, September 2026). Alignment and Claude’s Constitution, alignment training keeping up with capability.
• Measurements for understanding the pace of AI development inside frontier labs (Marina Favaro and Phillie Wright, Anthropic Institute, September 2026). The 26 percent figure and the definition of “leads,” the 100,000 weekly flags, classifier review, and roughly 50 escalations to human review.
• Ironies of Automation (Lisanne Bainbridge, Automatica 19(6), 775 to 779, 1983). The impossible task of real-time monitoring; decisions made at a rate the operator can follow.
• The flaws of policies requiring human oversight of government algorithms (Ben Green, Computer Law and Security Review 45, 105681, 2022). Survey of 41 policies; the shift to institutional oversight.
• Nate Soares (Wikipedia). President of the Machine Intelligence Research Institute; co-author of If Anyone Builds It, Everyone Dies (2025).
• Humans First: AI Isn’t the Revolution. We Are. (bosio.digital). “It processes information, not meaning.”

Frequently Asked Questions