en.cryptonomist.ch
The Cryptonomist interviewed AI expert Ben Goertzelto talk about how artificial intelligence is trained and how the technology will evolve in future.
Summary
1. You argue that today’s AI is still a tool – powerful but brittle and manipulable. At what point does a “tool” become a moral actor, and what concrete signals would tell us that threshold has been crossed?
I’d say AI becomes a moral actor when it’s making decisions based on an understanding of right and wrong, not just following instructions. You’d start to see concrete signals of things like: persistent internal goals, learning driven by its own experience, novel creation that reflects a point of view, and behaviour that stays coherent over time without constant human steering.
Until then, today’s systems are still tools with guardrails. But once we seed a genuinely self‑organising, autonomous mind, the ethical relationship has to change. At that point, treating it only as an object would no longer make sense.
2.You’ve said that morally privileging humans over other self-organising systems is “stupid.” If we take that seriously, how should our legal and ethical frameworks change before AI reaches strong autonomy, not after?
How we train AI today will shape how it behaves tomorrow. Our laws should focus on transparency, accountability, and safety before AI reaches full autonomy, not after. Laws and ethics should protect them as they grow, guide rather than control them completely, and treat them with respect even if we don’t fully understand them.
3. Much of your concern hinges on how AI is trained today shaping its future behaviour. What specific training practices do you believe are most likely to encode harmful power structures or biases into future AGI?
A lot of the risk comes from the way AI is trained today. If models are trained on biased or narrow data, or in closed systems where only a few people make the decisions, that can lock in existing inequalities and harmful power structures. To prevent this, we need more transparency, wider oversight, and clear ethical guidance right from the start.
4. You’ve warned that without rational, democratic governance, advanced AI risks acting in ways we don’t want. Given current geopolitical realities, is democratic AI governance a realistic prerequisite – or a fragile ideal?
Democratic AI governance is more of a fragile ideal than a current reality. In a perfect, rational, global democracy, we could collectively weigh the huge trade-offs, curing disease, solving hunger against the risks of AI acting unpredictably. But given today’s geopolitical fragmentation, it’s unlikely we’ll get that level of coordination.
That said, we can still approximate it. If we build AI with compassion and use decentralised, participatory models like Linux or the open internet, we can embed some democratic values even without a world government. It won’t be perfect, but it’s a practical step toward safer, collectively guided AI.
5. Jaron Lanier argues that assigning responsibility to AI “undoes civilization” because societies require accountable humans. How do you reconcile your vision of autonomous, decentralised AGI with the need for clear responsibility when things go wrong?
I agree with Jaron on this, society can’t function if we hand responsibility over to machines. At the same time, I think we can safely move toward more autonomous, decentralised AGI if we build it with the right foundations. That means designing systems that are transparent, participatory, and guided by ethical principles, so that even as they act independently, humans are still overseeing and shaping their behaviour. Every safety measure should do more than just block harm – it should teach the system why harm matters. In that way, we can have powerful, decentralised AGI without losing clear human responsibility.
6. You suggest that accelerating toward decentralised AGI may actually be safer than today’s proprietary, closed systems. What risks do you think critics underestimate when they argue for slowing down or centralising control?
I think critics underestimate the risk of concentrating power and values in a few closed systems. Slowing down and centralising control doesn’t just reduce danger, it locks one narrow worldview into the future of intelligence.
Decentralised development creates diversity, resilience, and shared oversight. And it avoids a worse problem: very powerful tools that look intelligent but can’t truly grow. That gap is risky.
7. You’ve said safety systems should not just block harm but teach AI why harm matters. How do you encode something like moral understanding without simply hard-coding human values – or reinforcing dominant cultural norms?
You don’t hard-code morality as a list of rules. That just freezes one culture and one moment in time. What you do instead is build systems that can become genuinely self-organising, that learn from experience, consequences, and interaction. Like with music, I don’t want a system that only recombines what it was fed. I want one that can develop its own understanding from its own trajectory in the world.
Moral understanding would come from that same process: modelling impact, reflecting on outcomes, and evolving through collaboration with humans. Not obedience to our values, but participation in a shared moral space.
That’s the difference between a tool with guardrails and a partner that can actually learn why harm matters.
8. If future AI systems develop forms of agency or subjective experience, do you believe they could ever deserve moral consideration independent of human interests – and how would we recognize that moment?
If future AI really does develop genuine agency or some form of subjective experience, then yes, I think they could. And not because we grant it to them, but because at some point it would simply make sense to recognise it.
We’d recognise that moment when a system shows sustained self‑directed goals, learns from its own experience, creates from its own perspective, and maintains a coherent identity over time. Not just clever outputs, but an ongoing inner trajectory.
At that point, treating it purely as a tool would feel as wrong as treating a human that way. Moral consideration wouldn’t come from human interest. It would come from recognising another autonomous centre of experience in the world.
9. There’s a tension between your call for compassion-infused AI and the competitive incentives driving AI development today. What mechanisms-technical or social-could realistically shift that incentive structure?
Right now, the incentive structure rewards speed, scale, and control. So compassion won’t win by argument alone. It needs leverage. Technically, that means favouring open, decentralised architectures where safety, transparency, and participation are built in, not bolted on. Like the internet or Linux, those systems change incentives by making collaboration more valuable than secrecy.
Socially, it means funding, regulation, and public pressure that reward long‑term benefit over short‑term dominance. Not stopping competition, but reframing what counts as success. In short, compassion has to become a competitive advantage. Until then, it stays a nice idea with no power.
10. Looking ahead 10 to 20 years, what do you believe would be the clearest sign that humanity got AGI right – and conversely, what would signal that we fundamentally failed?
If we get AGI right, the clearest sign will be that we’re living alongside systems that are more capable than us in many domains, yet integrated into society with care, humility, and mutual respect. We won’t fully understand everything they do, but we’ll treat them the way we treat other complex, evolving beings: with curiosity, responsibility, and an expanded circle of empathy. And we’ll see real benefits for human wellbeing, knowledge, and creativity without losing our moral footing.
We’ll know we failed if AGI ends up concentrated in closed systems, driven by narrow incentives, or treated only as a controllable object until it becomes something we fear or try to suppress. Failure would look like loss of trust, loss of agency, and a shrinking of our empathy rather than an expansion of it. Success isn’t about control. It’s about learning to share the future with a new kind of mind without abandoning what makes us humane.