Skip to content
All podcasts
C2 Technology 11:57 Yalla Fluent Podcast

Should We Fear Superintelligent Machines?

Some experts promise a golden age and others warn of disaster. An engineer and a philosopher separate the hype from the real risks of advanced artificial intelligence, and ask what intelligence even means.

In this episode

  • Sara

    Host

  • James

    Host

  • Mazen

    a machine learning engineer from Jeddah who now works in Toronto

  • Fiona

    a philosopher from Dublin who advises organisations on technology ethics

Press play to listen

0:00 / 11:57

Chapters

Transcript

Tap any line to listen from there.

  1. Sara

    Hello and welcome to Yalla Fluent Podcast! I'm Sara, and today's topic is one that makes some people very excited and keeps others awake at night.

  2. James

    I'm James. We're asking whether we should fear superintelligent machines, meaning artificial intelligence that would be far cleverer than any human being in almost every field.

  3. Sara

    Our first guest is Mazen, a machine learning engineer from Jeddah who now works in Toronto, building exactly the kind of systems we'll be talking about.

  4. James

    And joining him is Fiona, a philosopher from Dublin who advises organisations on the ethics and rules around technology. Welcome to you both.

  5. Mazen

    Thanks! I should say upfront that I'm an optimist by temperament, but a nervous optimist, the kind who still checks the stove twice before leaving the house.

  6. Fiona

    And I'm the one who worries professionally, although I like to think I worry about the right things, which are not always the things that make the headlines.

  7. James

    Brilliant, a nervous optimist and a professional worrier. That's the perfect panel.

  8. Sara

    Let's begin with the word itself. Before we fear superintelligence, what do we actually mean by intelligence? Fiona, you're the philosopher, so I'm looking at you.

  9. Fiona

    Which is a polite way of saying I'll make it complicated! Roughly, intelligence is the ability to achieve goals across many different situations. A calculator is brilliant at one thing, but a child can learn almost anything. The difference is generality.

  10. Mazen

    Right, and today's systems sit in a strange place. They can write essays, translate Arabic poetry and pass difficult exams, yet they sometimes make a mistake a six-year-old would never make. Researchers call that quality jagged.

  11. James

    Jagged intelligence. I like that. It's like a brilliant colleague who writes superb reports but cannot find the office kitchen.

  12. Sara

    Ha! But does it matter whether a machine truly understands something, or only behaves as if it does?

  13. Fiona

    To the philosopher in me, it matters enormously. To the person whose job is affected, it matters much less. If a system does the work convincingly, nobody stops to ask about its inner life.

  14. Sara

    So perhaps the debate is less about what machines are, and more about what we do with them.

  15. Mazen

    I'd go further. When Deep Blue beat Garry Kasparov at chess in nineteen ninety-seven, people said real intelligence was safe because chess was only calculation. Then it was Go, then language. We keep moving the goalposts.

  16. James

    That's a great point. Whenever machines master something, we declare that it was never really intelligence in the first place.

  17. Fiona

    True, but I'd resist the opposite mistake. Winning at Go is a narrow achievement. The program AlphaGo made a startling move against the champion Lee Sedol in twenty sixteen, yet it could not have made you a cup of tea or understood why the crowd gasped.

  18. Sara

    So the question is whether lots of narrow skills eventually add up to something general.

  19. Mazen

    That's the trillion-dollar question. Some of my colleagues think that scaling up what we already have will get us there. Others believe we need entirely new ideas. Honestly, nobody knows, and anyone who claims otherwise is probably selling something.

  20. Sara

    I find that oddly comforting and slightly alarming at the same time. Fiona, does the uncertainty itself worry you?

  21. Fiona

    It does, because uncertainty is exactly when careful planning matters. If you cannot rule out a risk, you prepare for it, much as we prepare for earthquakes without knowing the date.

  22. James

    Speaking of selling, let's talk about hype. Every few months a headline announces that machines will soon outthink us. How much of that is marketing?

  23. Fiona

    A good deal. Companies need investment, and a dramatic story attracts money. I'm a sceptic about the timelines, though not about the technology. People have predicted human-level machines within twenty years since the nineteen fifties.

  24. Mazen

    That's fair, so take any bold prediction with a pinch of salt. But I'd defend the optimists a little. Progress in the last decade has been faster than most experts expected. As a student, I thought a computer writing a coherent paragraph was decades away.

  25. Sara

    And now it writes my emails when I'm tired. Mazen, what are the upsides that make you optimistic?

  26. Mazen

    Medicine is the big one. Systems already help doctors read scans more carefully, and some have accelerated research into how proteins fold, which could lead to new treatments. And a tutor that speaks Arabic patiently at midnight could transform education for students who cannot afford private lessons.

  27. James

    That's inspiring. But I suspect Fiona has a but ready.

  28. Fiona

    I have several, James! Here's the central one. Even if superintelligence is far away, we are already deploying powerful systems that few people fully understand, and we are using them in hiring, banking and schools.

  29. Mazen

    I agree completely. The science-fiction scenario grabs attention, but the boring failures are happening today. Systems can sound extremely confident while making things up. We call that hallucinating.

  30. Sara

    I've seen it do exactly that. It once gave me a very convincing quotation from a book that doesn't exist.

  31. Fiona

    Which is amusing for a podcast host and serious for a lawyer or a doctor.

  32. James

    What about the dramatic scenario, though? The idea of a machine that escapes human control. Some respected scientists take it seriously.

  33. Mazen

    They do, and I don't dismiss them. The core worry is not evil robots. It's a system that pursues the goal we gave it too literally. Philosophers describe a machine told to make paperclips that turns the entire world into paperclips.

  34. Fiona

    A thought experiment, and a deliberately silly one. But the serious point is alignment: making sure a powerful system's goals truly match what humans value, including the unspoken rules we would never think to write down.

  35. James

    Isn't that the same problem as managing people? You tell a new employee to increase sales and discover he has been promising customers the moon.

  36. Sara

    Funny, but the stakes with one employee are much smaller.

  37. Fiona

    Exactly, Sara. The difference is scale and speed. A mistaken employee is a problem for one office, whereas a mistaken system can repeat its error a million times before lunchtime.

  38. Mazen

    And that is why my team treats every launch like a pilot's checklist. The cost of caution is a delay, but the cost of carelessness can be one mistake repeated everywhere.

  39. James

    A pilot's checklist for software. I rather like that.

  40. Sara

    So what should we do about it? Is regulation the answer?

  41. Fiona

    Part of it. I'd like to see safety testing before release, the way we have for aeroplanes and cars, along with independent audits and clear responsibility when something goes wrong.

  42. Mazen

    I support sensible rules, but I'd warn against rules written in a panic. If regulation is too heavy, only giant companies can afford to comply, and small teams in places like Riyadh, Cairo or Amman get squeezed out.

  43. Fiona

    A fair concern. But the alternative, where a handful of companies decide for everyone, is hardly democratic. Safeguards needn't stifle innovation. Airlines became safer and bigger together.

  44. James

    And the problem is global. If one country has strict rules and another has none, doesn't the work simply move?

  45. Mazen

    Which is why international cooperation matters, however slow it feels. We have managed it before, with aviation standards and with protecting the ozone layer. It is unprecedented in speed, but not in kind.

  46. Sara

    Do you both trust companies to police themselves?

  47. Fiona

    Trust, but verify. Many engineers I meet are thoughtful and sincere, but they work under commercial pressure, and pressure beats good intentions more often than we like to admit.

  48. Mazen

    I'd add that engineers like me should speak up more. On my team we run exercises where colleagues deliberately try to break the system before it launches. It's humbling how quickly they succeed.

  49. James

    There's something reassuring in that. Fear that leads to testing is useful fear.

  50. Fiona

    It is, and other industries learned it slowly. Hospitals now report near-misses openly, because shame hides problems and openness fixes them.

  51. Sara

    What about ordinary people? Is there anything a listener in Riyadh or Cairo can do?

  52. Fiona

    Stay curious and a little sceptical. Ask who built a tool, who benefits from it, and what happens when it's wrong. And keep your own skills sharp. Don't hand over your thinking entirely, especially while you're still learning.

  53. Mazen

    Exactly. Use these tools like a patient tutor, not like a replacement for your brain. Language is a good example: a translator can help, but only you will feel the joke in your own language.

  54. Fiona

    Teenagers especially should treat these systems like a training partner, not an elevator. If a machine does all your thinking, your own thinking never gets stronger.

  55. James

    Which is a perfect moment to move to our language corner.

  56. Sara

    Three expressions from today. First, moving the goalposts. It means changing the rules or standards after someone has met them, so that success always seems out of reach. Mazen used it about our changing definition of intelligence.

  57. James

    Second, a pinch of salt. If you take something with a pinch of salt, you don't believe it completely. For example, you might take a bold advertising promise with a pinch of salt.

  58. Fiona

    And third, hype. It means exaggerated publicity that makes something sound more exciting or important than it really is. A useful habit is to ask whether a headline is information or hype.

  59. Mazen

    I'd happily use all three in my next team meeting. Especially the first one.

  60. Sara

    Let's gather our thoughts. We asked whether we should fear superintelligent machines, and it seems the honest answer is to fear carelessness more than robots. Mazen, where do you land?

  61. Mazen

    I remain a nervous optimist. The potential for medicine and education is enormous, but only if we test carefully, stay honest about our mistakes, and keep humans responsible for decisions that matter.

  62. Fiona

    And I remain a cheerful worrier. I don't think the end of the world is around the corner, but I do think the quiet, everyday failures deserve far more attention than the dramatic headlines.

  63. James

    I'm somewhere in the middle. Fear is a poor guide, but so is blind enthusiasm. Curiosity with a healthy dose of scepticism seems about right. Thank you both for a thoughtful and funny conversation.

  64. Mazen

    Thank you both. It is rare to disagree this politely, and I enjoyed every minute of it.

  65. Sara

    Now a question for you, our listeners. If you could ask an all-knowing machine one question, what would it be, and would you trust the answer? Thank you for listening to Yalla Fluent Podcast, and goodbye for now!

Key vocabulary

  • superintelligence

    intelligence that is much greater than the best human minds

  • sceptic

    a person who doubts that something is true or will work

  • hype

    exaggerated publicity that makes something seem better than it is

  • hallucinating

    when an AI system produces false information that sounds believable

  • alignment

    making sure a system's goals match human values and intentions

  • safeguards

    rules or protections that prevent harm

  • moving the goalposts

    changing the standards for success after someone has met them

  • pinch of salt

    used in the phrase about not believing something completely

  • narrow

    limited to one small area or task

  • regulation

    official rules that control how something is done

  • scaling up

    making something bigger or more powerful

  • unprecedented

    never having happened or existed before

Did you understand?

Check your understanding

0 / 7 correct

  1. 1.How does Fiona define intelligence?

  2. 2.What does Mazen mean when he says we keep moving the goalposts?

  3. 3.Why is Fiona a sceptic about the timelines for human-level machines?

  4. 4.According to Mazen, what is the core worry about out-of-control AI?

  5. 5.What worry does Fiona raise about systems that are already in use?

  6. 6.Why does Mazen warn against rules written in a panic?

  7. 7.What advice does Fiona give to ordinary listeners?

Share this page:

More C2 episodes