The Bomb and the Black Box

"Mark my words, AI is far more dangerous than nukes."

Elon Musk is indeed known for his many rhetorical outbursts and bold claims; after all this is the guy who told us that our cars would be driving themselves since 2015. So you’d be forgiven for not putting too much stock into this opinion. (Elon has since launched xAI, so his own fears have either lessened or he’s openly courting the danger he warned about)

Even so, Elon is but one of many prominent technologists who have drawn the comparison, and there are those that are still raising concerns today while others chase the AI frenzy. Just as the bomb – with its eerie flash, ominous mushroom cloud, and invisible radiation – captivated and terrified a generation, AI is quickly becoming this generation’s existential obsession. Fears of mass unemployment, cybersecurity exploits, and autonomized warfare grow by the year.

In reading The Thinking Machine, by Stephen Witt, this focus on AI's future risk, and its builders’ roles in acknowledging it, emerged as a core theme. Witt profiles NVIDIA and its founder Jensen Huang, and in doing so finds himself grappling with the onslaught of AI, propelled forward at light speed by NVIDIA’s cutting-edge chip designs and software stack. When it comes to the impact of AI on society, Jensen is unapologetic and stubbornly optimistic; those who read the tea leaves the opposite way are overly cynical or naïve.

Musk’s sentiment about the dangers of AI, quoted in Witt’s book, jumped off the page for me. Last year, I spent much of my reading time working my way through Richard Rhoades’ brilliant The Making of the Atomic Bomb. The 750-page tome thoroughly charts the creation of the bomb, starting first with developments in chemistry and physics in the early 20th century that would later make the science possible. It is a story of large characters, huge national ambitions and deep principles and philosophies that would foreshadow the bomb’s impact even before it was used.

The contexts surrounding AI and nuclear weapons are obviously distinct, but the parallel in charting the development of the technologies and their profoundly troublesome implications for humanity seems of obvious importance to me through this lens.

The first challenge is that the horrible impacts of the atomic bomb were understood (once they were believed) instantly. The impacts of AI are far less certain or agreed upon. When pressed on the existential risk of AI, Huang is unapologetically dismissive and believes the concerns are more sci-fi than reality. The author does a good job of teasing this out of him, up until the point at the end of the book where Huang brutally berates him for endlessly bothering him with this topic and wasting his time. Huang’s anger, the author poses, is strategic; to the reader, you can't help but feel that Witt touched a nerve, and that Huang's belief in the technology is overly certain and missing a shade of nuance (perhaps blinded by profit).

What is clear is that some of AI’s greatest minds, people like computer scientist Geoffrey Hinton interviewed in the book, are sounding the alarm. Not everyone has made a direct comparison with nuclear weapons, but the concerns are extreme. Agentic AI, one of the key themes reshaping how users interact with the technology today has been highlighted as potentially “uncontrollable” by prominent AI scientists Max Tegmark and Yoshua Bengio.

As Witt reveals, what is missing from NVIDIA and Huang is any instinct for regulation or contemplation about the risks of AI. Perhaps Jensen's claim is fair - that speculation about existential risk is not supported by the facts; there certainly are a number of steps between today's AI chips and the end of humanity. But perhaps Huang's dismissiveness of existential risk as a sci-fi trope reveals not just a lack of imagination, but a lack of motivation to be imaginative.

The silence from the $5 trillion company (by market cap) is revealing. Witt notes briefly that when it came to crypto mining and the unexpected utility of NVIDIA chips, the company quietly embraced the market and designed a custom chip for crypto miners.

To be sure there was a notable lack of enthusiasm, to be sure, but the shareholder motivation was clear. Similarly, with recent dialogs around China, Huang has been at odds with geopolitics in promoting his chips in China. Neither Republicans, Democrats, or even the Chinese government want NVIDIA to have anything to do with the Chinese market. But the profit motive is strong, and Huang has lobbied both the US and Chinese governments heavily to keep NVIDIA relevant.

Today, there are those who wish we could simply stop building AI, or at least slow down. Ironically, when the idea of the bomb was growing into reality among the world's scientists and military leaders, the key question was who could get their first. In this way, Huang is carrying on the legacy of his scientific predecessors. If someone, or some country, is to control the power that AI will unleash, it should be him and the United States. Nothing in The Thinking Machine reveals such a deliberate calculus on the part of Huang; what it does a reveal is a desire to avoid the question altogether.

Of course, the overarching narrative behind the bomb was World War II. Despite the horrific new paradigm that nuclear weapons birthed, their conception is looked back as necessary by many, even heroic. There is no admirable parallel to today. It is absolutely possible that AI will experience a redemption arc, with the technology revolutionizing medicine and discovery for the betterment of humanity.  Those achievements are for now overshadowed by cynical desires for corporate downsizing, shareholder maximization, and dulling the faculties of the general populace.

We cannot yet say definitively whether AI will be an existential risk to humanity, but the question is critical to interrogate. Reading The Making of the Atomic Bomb revealed that the core personalities involved with the bomb effort interrogated their roles, their mission, and the morality of what they were building deeply. If anything, I came away immensely impressed at the character of the individuals involved in such an existential endeavor. It was a small modicum of hope in an often hopeless backdrop that men and women of good character were at the center of a new and terrible world.

Looking around at today's landscape of tech leaders and politicians, can anyone have any true hope that we are led by people of character who understand the implications of what they are building?