Should We Fear Superintelligent Machines?
Some experts promise a golden age and others warn of disaster. An engineer and a philosopher separate the hype from the real risks of advanced artificial intelligence, and ask what intelligence even means.
In this episode
Sara
Host
James
Host
Mazen
a machine learning engineer from Jeddah who now works in Toronto
Fiona
a philosopher from Dublin who advises organisations on technology ethics
Press play to listen
0:00 / 11:57
Chapters
Transcript
Tap any line to listen from there.
- Sara
Hello and welcome to Yalla Fluent Podcast! I'm Sara, and today's topic is one that makes some people very excited and keeps others awake at night.
- James
I'm James. We're asking whether we should fear superintelligent machines, meaning artificial intelligence that would be far cleverer than any human being in almost every field.
- Sara
Our first guest is Mazen, a machine learning engineer from Jeddah who now works in Toronto, building exactly the kind of systems we'll be talking about.
- James
And joining him is Fiona, a philosopher from Dublin who advises organisations on the ethics and rules around technology. Welcome to you both.
- Mazen
Thanks! I should say upfront that I'm an optimist by temperament, but a nervous optimist, the kind who still checks the stove twice before leaving the house.
- Fiona
And I'm the one who worries professionally, although I like to think I worry about the right things, which are not always the things that make the headlines.
- James
Brilliant, a nervous optimist and a professional worrier. That's the perfect panel.
- Sara
Let's begin with the word itself. Before we fear superintelligence, what do we actually mean by intelligence? Fiona, you're the philosopher, so I'm looking at you.
- Fiona
Which is a polite way of saying I'll make it complicated! Roughly, intelligence is the ability to achieve goals across many different situations. A calculator is brilliant at one thing, but a child can learn almost anything. The difference is generality.
- Mazen
Right, and today's systems sit in a strange place. They can write essays, translate Arabic poetry and pass difficult exams, yet they sometimes make a mistake a six-year-old would never make. Researchers call that quality jagged.
- James
Jagged intelligence. I like that. It's like a brilliant colleague who writes superb reports but cannot find the office kitchen.
- Sara
Ha! But does it matter whether a machine truly understands something, or only behaves as if it does?
- Fiona
To the philosopher in me, it matters enormously. To the person whose job is affected, it matters much less. If a system does the work convincingly, nobody stops to ask about its inner life.
- Sara
So perhaps the debate is less about what machines are, and more about what we do with them.
- Mazen
I'd go further. When Deep Blue beat Garry Kasparov at chess in nineteen ninety-seven, people said real intelligence was safe because chess was only calculation. Then it was Go, then language. We keep moving the goalposts.
- James
That's a great point. Whenever machines master something, we declare that it was never really intelligence in the first place.
- Fiona
True, but I'd resist the opposite mistake. Winning at Go is a narrow achievement. The program AlphaGo made a startling move against the champion Lee Sedol in twenty sixteen, yet it could not have made you a cup of tea or understood why the crowd gasped.
- Sara
So the question is whether lots of narrow skills eventually add up to something general.
- Mazen
That's the trillion-dollar question. Some of my colleagues think that scaling up what we already have will get us there. Others believe we need entirely new ideas. Honestly, nobody knows, and anyone who claims otherwise is probably selling something.
- Sara
I find that oddly comforting and slightly alarming at the same time. Fiona, does the uncertainty itself worry you?
- Fiona
It does, because uncertainty is exactly when careful planning matters. If you cannot rule out a risk, you prepare for it, much as we prepare for earthquakes without knowing the date.
- James
Speaking of selling, let's talk about hype. Every few months a headline announces that machines will soon outthink us. How much of that is marketing?
- Fiona
A good deal. Companies need investment, and a dramatic story attracts money. I'm a sceptic about the timelines, though not about the technology. People have predicted human-level machines within twenty years since the nineteen fifties.
- Mazen
That's fair, so take any bold prediction with a pinch of salt. But I'd defend the optimists a little. Progress in the last decade has been faster than most experts expected. As a student, I thought a computer writing a coherent paragraph was decades away.
- Sara
And now it writes my emails when I'm tired. Mazen, what are the upsides that make you optimistic?
- Mazen
Medicine is the big one. Systems already help doctors read scans more carefully, and some have accelerated research into how proteins fold, which could lead to new treatments. And a tutor that speaks Arabic patiently at midnight could transform education for students who cannot afford private lessons.
- James
That's inspiring. But I suspect Fiona has a but ready.
- Fiona
I have several, James! Here's the central one. Even if superintelligence is far away, we are already deploying powerful systems that few people fully understand, and we are using them in hiring, banking and schools.
- Mazen
I agree completely. The science-fiction scenario grabs attention, but the boring failures are happening today. Systems can sound extremely confident while making things up. We call that hallucinating.
- Sara
I've seen it do exactly that. It once gave me a very convincing quotation from a book that doesn't exist.
- Fiona
Which is amusing for a podcast host and serious for a lawyer or a doctor.
- James
What about the dramatic scenario, though? The idea of a machine that escapes human control. Some respected scientists take it seriously.
- Mazen
They do, and I don't dismiss them. The core worry is not evil robots. It's a system that pursues the goal we gave it too literally. Philosophers describe a machine told to make paperclips that turns the entire world into paperclips.
- Fiona
A thought experiment, and a deliberately silly one. But the serious point is alignment: making sure a powerful system's goals truly match what humans value, including the unspoken rules we would never think to write down.
- James
Isn't that the same problem as managing people? You tell a new employee to increase sales and discover he has been promising customers the moon.
- Sara
Funny, but the stakes with one employee are much smaller.
- Fiona
Exactly, Sara. The difference is scale and speed. A mistaken employee is a problem for one office, whereas a mistaken system can repeat its error a million times before lunchtime.
- Mazen
And that is why my team treats every launch like a pilot's checklist. The cost of caution is a delay, but the cost of carelessness can be one mistake repeated everywhere.
- James
A pilot's checklist for software. I rather like that.
- Sara
So what should we do about it? Is regulation the answer?
- Fiona
Part of it. I'd like to see safety testing before release, the way we have for aeroplanes and cars, along with independent audits and clear responsibility when something goes wrong.
- Mazen
I support sensible rules, but I'd warn against rules written in a panic. If regulation is too heavy, only giant companies can afford to comply, and small teams in places like Riyadh, Cairo or Amman get squeezed out.
- Fiona
A fair concern. But the alternative, where a handful of companies decide for everyone, is hardly democratic. Safeguards needn't stifle innovation. Airlines became safer and bigger together.
- James
And the problem is global. If one country has strict rules and another has none, doesn't the work simply move?
- Mazen
Which is why international cooperation matters, however slow it feels. We have managed it before, with aviation standards and with protecting the ozone layer. It is unprecedented in speed, but not in kind.
- Sara
Do you both trust companies to police themselves?
- Fiona
Trust, but verify. Many engineers I meet are thoughtful and sincere, but they work under commercial pressure, and pressure beats good intentions more often than we like to admit.
- Mazen
I'd add that engineers like me should speak up more. On my team we run exercises where colleagues deliberately try to break the system before it launches. It's humbling how quickly they succeed.
- James
There's something reassuring in that. Fear that leads to testing is useful fear.
- Fiona
It is, and other industries learned it slowly. Hospitals now report near-misses openly, because shame hides problems and openness fixes them.
- Sara
What about ordinary people? Is there anything a listener in Riyadh or Cairo can do?
- Fiona
Stay curious and a little sceptical. Ask who built a tool, who benefits from it, and what happens when it's wrong. And keep your own skills sharp. Don't hand over your thinking entirely, especially while you're still learning.
- Mazen
Exactly. Use these tools like a patient tutor, not like a replacement for your brain. Language is a good example: a translator can help, but only you will feel the joke in your own language.
- Fiona
Teenagers especially should treat these systems like a training partner, not an elevator. If a machine does all your thinking, your own thinking never gets stronger.
- James
Which is a perfect moment to move to our language corner.
- Sara
Three expressions from today. First, moving the goalposts. It means changing the rules or standards after someone has met them, so that success always seems out of reach. Mazen used it about our changing definition of intelligence.
- James
Second, a pinch of salt. If you take something with a pinch of salt, you don't believe it completely. For example, you might take a bold advertising promise with a pinch of salt.
- Fiona
And third, hype. It means exaggerated publicity that makes something sound more exciting or important than it really is. A useful habit is to ask whether a headline is information or hype.
- Mazen
I'd happily use all three in my next team meeting. Especially the first one.
- Sara
Let's gather our thoughts. We asked whether we should fear superintelligent machines, and it seems the honest answer is to fear carelessness more than robots. Mazen, where do you land?
- Mazen
I remain a nervous optimist. The potential for medicine and education is enormous, but only if we test carefully, stay honest about our mistakes, and keep humans responsible for decisions that matter.
- Fiona
And I remain a cheerful worrier. I don't think the end of the world is around the corner, but I do think the quiet, everyday failures deserve far more attention than the dramatic headlines.
- James
I'm somewhere in the middle. Fear is a poor guide, but so is blind enthusiasm. Curiosity with a healthy dose of scepticism seems about right. Thank you both for a thoughtful and funny conversation.
- Mazen
Thank you both. It is rare to disagree this politely, and I enjoyed every minute of it.
- Sara
Now a question for you, our listeners. If you could ask an all-knowing machine one question, what would it be, and would you trust the answer? Thank you for listening to Yalla Fluent Podcast, and goodbye for now!
Key vocabulary
superintelligence
intelligence that is much greater than the best human minds
sceptic
a person who doubts that something is true or will work
hype
exaggerated publicity that makes something seem better than it is
hallucinating
when an AI system produces false information that sounds believable
alignment
making sure a system's goals match human values and intentions
safeguards
rules or protections that prevent harm
moving the goalposts
changing the standards for success after someone has met them
pinch of salt
used in the phrase about not believing something completely
narrow
limited to one small area or task
regulation
official rules that control how something is done
scaling up
making something bigger or more powerful
unprecedented
never having happened or existed before
Did you understand?
Check your understanding
0 / 7 correct
1.How does Fiona define intelligence?
2.What does Mazen mean when he says we keep moving the goalposts?
3.Why is Fiona a sceptic about the timelines for human-level machines?
4.According to Mazen, what is the core worry about out-of-control AI?
5.What worry does Fiona raise about systems that are already in use?
6.Why does Mazen warn against rules written in a panic?
7.What advice does Fiona give to ordinary listeners?