Q: In Amodei’s recent proposal to better regulate AI, there is not much (but please check) about allowing AI agents to assume or get a legal person status such as natural persons and corporations can. What do you think about that, also in the light of our ‘paperclip’ discussion at AI can build factories – Fred Lafeber . Would it be a sensible measure not to allow AI agents to independently acquire land, capital, factories etc or would that be hard to monitor and easy to evade?
A: I checked Amodei’s new proposal. Your impression is correct: “We Must Pace the Frontier” says essentially nothing about AI legal personhood, property ownership, corporate ownership or an AI’s capacity to hold assets. Searches of the essay for personhood, property, corporation, ownership and even legal produce no discussion of the issue. His regulatory focus is instead on embedded independent evaluators, capability-dependent safety requirements, coordination among frontier labs, and ultimately international limits on things such as recursive self-improvement.
I think you have identified a real missing layer in the present AI-safety debate, although I would formulate the rule somewhat differently from simply “AI must never have legal personhood.”
Your factory argument becomes much stronger here
In your factory discussion, the crucial objection to the paperclip scenario was very physical and institutional. An AI cannot just think:
I need more paperclips → therefore I shall build a factory.
It needs capital, land, contracts, electricity, planning permission, suppliers, machinery, bank accounts, people or robots and access to the legal system. Your article therefore concluded that, for quite some time, humans and institutions remain essential intermediaries.
That is potentially a very powerful safety barrier.
And it suggests a regulatory principle that I think deserves more attention:
Don’t only regulate how intelligent an AI may become. Regulate how independently it may acquire and control resources in the real world.
That could prove easier to enforce than trying to determine whether a model secretly has dangerous intentions.
But I wouldn’t simply prohibit “AI legal personhood”
There is a paradox here.
Suppose we say:
“An AI can never own property, establish a company or have a bank account.”
At first this sounds highly protective. But the AI can simply operate through:
AI → human nominee → corporation → bank account → land → factory.
Or:
AI → existing corporation → contracts → leased factory → robots.
It doesn’t actually need to own the factory.
It could lease equipment, hire contractors, purchase cloud computing, arrange loans, trade securities or cryptocurrencies, instruct a human-controlled corporation, or control the economic decisions of a company whose formal shareholder remains human.
So banning personhood alone might create an impressive legal wall with a rather large door beside it.
There is already an active legal debate about precisely this. Some scholars advocate an “A-corp”: humans must remain disclosed beneficial owners, while AI is allowed to operate the company. Their argument is almost the reverse of yours: giving the AI-operated enterprise a defined legal identity makes its assets visible, taxable, regulatable and seizable. Another recent legal analysis argues explicitly that legal personhood is neither necessary nor sufficient for controlling agentic AI—the core problem is whether law can identify and constrain whoever or whatever exercises effective control.
Interestingly, this is no longer entirely theoretical. Argentina’s government has already publicly defended the idea of a legal framework for AI-run companies, arguing that legal personality can make their assets and obligations identifiable.
So I wouldn’t make personhood itself the red line.
I would make independent control of resources the red line
For sufficiently capable autonomous AI, I think a precautionary regime might say:
- No AI may be the ultimate beneficial owner of significant assets. There must always be an identifiable human or human-governed legal entity behind it.
- AI may not independently acquire or exercise effective control over designated strategic resources—large compute clusters, critical infrastructure, substantial land, industrial plants, weapons production, major energy facilities, etc.—without accountable human authorization.
- AI agents conducting significant economic transactions must operate under a registered principal. Someone remains legally responsible for their actions.
- Above defined thresholds, humans must authorize expansion of the agent’s resource envelope: borrowing €1 billion, buying another data centre, acquiring a company, commissioning a factory, or multiplying its compute by 100 shouldn’t be something an agent can decide and execute entirely by itself.
There is already a useful analogy in European anti-money-laundering law. EU rules ultimately require a corporation’s beneficial owner to be identifiable as a natural person, even where ownership passes through layers of companies. The same concept could be extended from beneficial ownership to beneficial control by autonomous systems.
I’d call it something like an AI resource-control principle:
A sufficiently capable AI system may use economic resources delegated to it, but may not autonomously enlarge its own control over capital, compute, property, infrastructure or production capacity beyond predefined limits without authorization from an identifiable accountable human or institution.
That seems to me substantially stronger than simply saying “AI cannot be a legal person.”
Would it be easy to evade?
Partly.
Land and factories are actually relatively easy to police. They exist somewhere. Land has registries. Large factories require electricity, insurance, suppliers, licenses, construction contracts and banking. A hyperscale data centre cannot quietly disappear into cyberspace.
That supports your original intuition: the physical economy creates choke points.
Capital is harder. An autonomous system could control cryptocurrency wallets, create shell structures, hire people online, use offshore jurisdictions, lease resources instead of purchasing them or arrange transactions through intermediaries. And an AI does not need its name on the title deed to have effective control over an asset.
That means regulators should watch control rather than ownership.
Consider two situations:
A. Claude owns Factory Ltd legally, but every €10 million investment requires approval by a human supervisory board.
B. Fred owns 100% of Factory Ltd on paper, but an AI controls its bank accounts, procurement, production, robots, borrowing and investment strategy and Fred automatically signs whatever it recommends.
Legally A looks more autonomous.
Functionally B is much more dangerous.
That distinction strikes me as very important.
And this connects directly to our “why/how” discussion
It actually gives me a somewhat more optimistic view of existential risk.
We’ve been asking:
What happens if the AI’s “how” becomes so autonomous that it starts generating its own intermediate “whys”?
Your factory argument introduces another barrier:
thinking → deciding → acquiring resources → acting physically
Even if AI crosses the first two boundaries, society can deliberately preserve friction at the third.
That might be considerably more robust than trying to ensure that every extremely intelligent AI forever thinks exactly as we would like it to think.
So rather than relying exclusively on:
Make sure the superintelligence never wants anything dangerous.
we could also build a world in which:
Even extremely capable AI cannot independently acquire the compute, capital, energy, companies and physical infrastructure necessary to turn an unexpected objective into large-scale power.
Amodei is mainly trying to slow and supervise the development of the brain. Your idea is essentially to regulate the hands, wallet and property rights of that brain.
I think both are necessary.
And there is a slightly ironic conclusion. It may ultimately be less important whether AI is legally considered a “person” than whether we allow it to behave economically like one. An AI with no legal rights but autonomous control of €50 billion, data centres and factories worries me much more than an AI corporation with legal personality, €1 million in registered capital, transparent accounts and humans who remain accountable for increasing its authority.
Q: But what about the fact that Argentina does already allow it. Should we try to make an international agreement in exactly this field.
Yes — and after checking the Argentine position more carefully, I think this could be one of the more sensible areas for an international AI agreement, perhaps even more tractable than trying to agree internationally on how powerful models may become.
First, one correction to my previous answer. Argentina has not yet enacted a law allowing autonomous AI corporations. President Milei has explicitly proposed a new category of “non-human corporation,” operated by AI agents or robots, in which human shareholders would not necessarily be required. His government has submitted broader corporate-law reform to Congress, and Milei has publicly defended granting legal personality to AI-run firms. But the relevant proposals are still going through the Argentine legislature; they are not current law.
That actually makes the issue more timely rather than less.
Why an international agreement makes sense
Argentina illustrates precisely the problem. If one country creates a highly permissive regime, an autonomous AI need not persuade every country to recognize it. It could incorporate in Argentina and then try to operate internationally—rather like companies today choose Delaware, Luxembourg, Ireland or Singapore for particular legal or tax advantages.
Milei is quite explicit that Argentina wants to become an attractive jurisdiction for such entities. He argues that giving AI-run corporations legal personality would make them more accountable because their assets could be identified, sued and seized.
There is some merit in that argument. But I think it misses the issue we have been discussing.
The danger isn’t primarily:
AI corporation behaves badly → whom can we sue?
It is:
AI acquires capital → uses capital to acquire more compute → buys companies → acquires land → builds data centres or factories → uses the additional resources to acquire still more capability and independence.
Legal personality could remove important human friction from that process.
And once one country permits it, there is a strong argument for international coordination because otherwise we create regulatory arbitrage.
I would make the agreement narrower than “AI may never be a legal person”
I wouldn’t try to obtain an international convention saying AI can never have legal personality. That opens an enormous philosophical and legal debate about whether a future conscious AI might deserve rights.
Instead, regulate something much more concrete:
No autonomous AI system may independently acquire or expand control over substantial economic or physical resources without an identifiable and accountable human or human-governed institution authorizing that expansion.
That avoids the consciousness debate completely.
An international agreement could establish five basic rules:
- An AI cannot be the ultimate beneficial owner or ultimate controller of a corporation holding significant assets.
- An AI-run entity must always have an identifiable human or human-governed accountable principal.
- Above defined thresholds, AI cannot independently acquire land, companies, major amounts of capital, strategic compute, energy infrastructure, factories or critical infrastructure.
- Banks, corporate registries, land registries, securities markets and major compute providers must establish who ultimately controls an AI agent before providing access.
- Countries agree not to recognize foreign AI-controlled corporations that don’t meet those requirements.
The last point is particularly powerful.
Suppose Argentina eventually says:
“AI-247 Ltd is a perfectly legal Argentine autonomous corporation.”
Denmark, the EU, US, UK and Japan could nevertheless say:
“Fine. But AI-247 Ltd cannot open a bank account here, purchase a Danish company, acquire land, obtain a 500-MW electricity connection or operate a large compute cluster unless an accountable human principal is identified.”
Suddenly the Argentine loophole becomes much less valuable.
- In fact, this may be easier to enforce than controlling intelligence
This connects strongly with your factory argument.
It is extraordinarily difficult to regulate something like:
“An AI must never formulate a dangerous intermediate objective.”
How would we know?
But societies already have institutions designed to monitor:
bank accounts, company ownership, land ownership, electricity use, securities transactions, large loans, construction permits, industrial facilities and major data centres.
So instead of trying to see inside the AI’s mind, we regulate the points at which intelligence is converted into power.
That strikes me as a surprisingly robust idea.
The Council of Europe’s existing Framework Convention on AI doesn’t really do this. It establishes broad obligations concerning human rights, democracy, transparency, accountability and risk management, but it doesn’t establish an international regime governing AI ownership of economic resources or AI-controlled corporations. It was deliberately designed as a broader framework that can later be supplemented by more specific instruments.
A protocol dealing specifically with autonomous agents, legal capacity and resource control could therefore fit logically alongside it.
Argentina doesn’t necessarily have to join
This is another important point.
You don’t initially need all 190-plus countries.
If the EU, US, UK, Canada, Japan, South Korea, Australia and several major financial centres agreed that autonomous AI corporations without accountable human controllers would not be recognized within their jurisdictions, that would cover an enormous share of world capital, advanced semiconductors, cloud computing and financial markets.
It would resemble anti-money-laundering rules: countries retain their own company law, but accessing the international financial system requires compliance with certain standards.
That is probably much more realistic than a grand UN treaty banning autonomous AI corporations everywhere.
And it gives our “why/how” discussion another layer
We originally had:
Human: supplies the why
AI: supplies the how
Then we realized that increasingly autonomous AI may begin creating intermediate whys:
To achieve X, I need Y.
To get Y, I need Z.
Therefore I should acquire Z.
Your idea effectively puts a legal barrier at a crucial next step:
AI: “Therefore I need €4 billion, a data centre and a factory.”
Law: “You cannot independently make that leap. An accountable human institution must authorize it.”
That doesn’t solve alignment.
But it prevents an intellectual objective from automatically turning into economic self-expansion.
And I increasingly think that may be one of the most overlooked safeguards in the existential-risk discussion.
There is also a nice inversion of Milei’s argument. He says giving autonomous AI companies legal personality brings them inside the law.
Your counterargument could be:
Yes, AI should operate inside the law—but that doesn’t mean the law should grant AI the same mechanisms humans invented to accumulate capital, acquire property and concentrate economic power.
Those are two completely different propositions.
I think this is strong enough that it could become a second short article following your “AI can build factories” piece: something like “Don’t give AI a wallet: why control over resources may matter more than control over intelligence.”
