In recent weeks, executives at some major AI companies have warned that the technology they are developing is dangerous. Anthropic CEO Dario Amodei said that progress should be slowed, while OpenAI CEO Sam Altman said he agreed that the pace of “frontier” work should be adjusted.
Jacob Coxon, a researcher who left Anthropic, accused Anthropic and OpenAI, where he had previously worked, of failing to act responsibly. Coxon wrote that those developing AI believe it could kill all of humanity by the end of the decade. Anthropic employee Evan Hubinger also shared this view, saying he put the likelihood at “>10%” over the next decade.
One concern is that AI could help design biological weapons such as deadly viruses targeting human genes, fungi that could destroy agriculture, or toxins that could be added to water without detection. In 2022, researchers at Collaborations Pharmaceuticals showed that a molecule generator they had developed produced 40,000 molecules that could be chemical warfare agents in less than six hours; some were designed to be more toxic than known nerve agents.
David Magnus of Stanford found this very frightening. According to Dunja Sabra, a biosecurity researcher at the University of Hamburg, AI bots can provide scientific information, experimental instructions and video tutorials. The spread of gene editing, synthetic biology and home laboratories increases the risk.
DNA screening, “red-teaming,” “blue-teaming” and AI safety measures are in place; however, Sabra says that a determined person could eventually succeed.
Why it matters
This discussion shows that AI safety is not limited to how systems behave, but also encompasses the possibility of the misuse of biological and chemical threats during the design phase. Although existing DNA screening, cross-team oversight methods and other safety measures aim to mitigate the risk, it remains unclear whether they are sufficient to prevent the efforts of a determined person. Therefore, the central question is not only how the pace of development should be adjusted, but also at what stage safety controls should be applied and to what extent they should be considered sufficient. The issue directly concerns not only executives and employees of AI companies, but also researchers working on gene editing, synthetic biology and home laboratories.
Term: agent
An AI agent is software that calls tools and carries out multi-step tasks to achieve a goal, rather than producing a single response.