Reports published by some of the world’s leading AI companies An unusual proposal On Tuesday, it demanded that companies be given their “right to be warned about advanced artificial intelligence.”
Who do they want to warn? you. The people. Anyone who will listen.
The 13 signatories are current and former employees of OpenAI and Google DeepMind. They believe that AI has great potential for good, but they worry that the technology could cause widespread harm if not properly safeguarded.
“I’m scared. I’d be crazy not to,” Daniel Cocotazlo, a signatory who left OpenAI in April after losing faith that the company’s leadership would manage its technology responsibly, told me this week. Several other security-conscious employees recently did the same. has gone, intensifying concerns that OpenAI is not taking the risks of the technology seriously enough. (Disclosure: Vox Media is one of several publishers that have signed partnership agreements with OpenAI. Our reporting is editorially independent.)
Understanding AI and the companies that build it
Artificial intelligence is poised to change the world from media to medicine and beyond – and Future Perfect has it covered.
- Leaked OpenAI documents reveal offensive tactics against ex-employees
- ChatGPT can talk, but the OpenAI staff can’t confirm
- “I’ve lost faith”: Why the OpenAI team tasked with saving humanity has crashed
- This article is OpenAI training data
It might be tempting to see the new proposal as just another open letter put out by “doomers” who want to put a break on AI because they worry it will go rogue and wipe out all of humanity. This is not all. The signatories share the concerns of both the “AI ethics” camp, which is more concerned about current AI harms such as racial bias and misinformation, and the “AI security” camp, which is more concerned about AI as a future existential risk.
These camps are sometimes pitted against each other. The new proposal aims to change the incentives of leading AI companies to make their operations more transparent to outsiders – and that would benefit everyone.
The signatories are calling on AI companies to voice their concerns about the technology – to company boards, regulators, independent expert bodies and, if necessary, directly to the public – without retaliation. Six of the signatories, including four current and two former OpenAI employees, are anonymous because of fear of retaliation against them. The proposal is backed by some of the biggest names in the field: Geoffrey Hinton (often called the “Godfather of AI”), Joshua Bengio and Stuart Russell.
To be clear, the signatories are not saying that they should be free to disclose their intellectual property or trade secrets, but that they want to be able to express concerns about risk as long as they protect them. To ensure whistleblowers are protected, they want companies to set up an anonymous process through which employees can report their concerns “to the company’s board, regulators and an appropriate independent body with relevant expertise.”
An OpenAI spokesperson told Vox that current and former employees already have forums for leadership to voice their thoughts during office hours, question-and-answer sessions with the board, and an anonymous forum. Integrity Hotline.
“General Whistleblower Protection [that exist under the law] Inadequate because they focus on illegal activity, where many of the risks we worry about are not yet controlled,” the signatories wrote in the proposal. They have retained a prominent lawyer, Lawrence Lessig, who previously advised Facebook whistleblower Frances Haugen and who was once a New Yorker. described As “the most important intellectual property thinker in the Internet age.”
Another of their demands: no more disparagement agreements barring company insiders from voicing risk-related concerns. Former OpenAI employees have long felt upset because, after leaving, the company signed offboarding agreements with non-disparagement provisions. After Vox reported about employees who felt pressured to sign or surrender their vested equity in the company, OpenAI said it was in the process of removing non-disclosure agreements.
These contracts were so unusually restrictive that they raised alarm bells even for employees who had left the company on good terms, such as Jacob Hilton, one of the signatories of the “right to warn” proposal. He wasn’t particularly worried about OpenAI’s security approach while working there, but when he left in early 2023 to do research elsewhere, the offboarding agreement worried him.
“It basically threatened to take away a large fraction of my compensation unless I signed an unnecessary and dishonorable contract,” Hilton told me. “I felt that applying these agreements so broadly would have a chilling effect on former employees’ ability to make reasonable criticisms.”
Ironically, it was OpenAI’s attempt to silence him that made him speak.
Hilton signed on to the new proposal, he said, because companies need to know that employees will call them when they talk about a big game about security in public — like OpenAI. did – Only then oppose that behind closed doors.
“Public promises are often written by company employees who really care, but then the company doesn’t have much incentive to stick to its promises if the public can’t find out. [about violations]” said Hilton. That’s where the new proposal comes in. “It’s about creating a framework where the company is encouraged to stick to its public promises.”
It’s about changing incentives for the entire AI industry
AI security researchers often worry about AI models being misaligned—pursuing goals in ways that are inconsistent with our standards. But you know what’s really hard to align? People. Especially when all the incentives are pushing them in the wrong direction.
Those who finish second in Silicon Valley are rarely remembered; First out of the gate is rewarded. The culture of competition means there is a strong incentive to build sophisticated AI systems quickly. And the profit imperative also means there’s a strong incentive to commercialize those systems and release them into the world.
OpenAI staff have increasingly noticed this. Jan Lake, who led the company’s alignment team until stepping down last month, said An X post that “safety culture and processes have taken a backseat to shiny products.”
Carol Wainwright, who served under Lake, resigned last week for similar reasons. “Over the last six months or so, I’ve become increasingly concerned that the incentives to do things in OpenAI are not well set,” he told me. “There are very, very strong incentives to increase profits that leadership has succumbed to some of those incentives at the expense of doing more mission-aligned work.”
So the big question is how can we change the underlying incentive structure that drives all actors in the AI industry?
For a while, the hope was that setting up AI organizations with unusual governance structures would do the trick. OpenAI, for example, started as a non-profit, with a board whose mission was not to keep shareholders happy but to protect humanity’s best interests. Wainwright said that’s part of why he was excited to work there: He thought the structure would keep incentives in place.
But OpenAI soon discovered that these days you need a ton of computing power to run large-scale AI experiments—more than that. 300,000 times It’s what you needed a decade ago – and it’s incredibly expensive. To stay on the cutting edge, it had to create a profitable arm and partner with Microsoft. OpenAI wasn’t alone in this: rival company Anthropic, whose former OpenAI staff wanted to focus more on security, began by arguing that we needed Change the incentive structure inherent in the industryWith profit incentives, but it joins forces with Amazon.
For a board tasked with protecting the best interests of humanity? That sounds great in theory, but OpenAI’s board drama last November — when the board tried to fire CEO Sam Altman only to quickly reinstate him — proves it doesn’t work.
“I think it showed that the board didn’t have the teeth that one might have expected,” Wainwright told me. “It makes me question how much the board can hold the organization accountable.”
Hence this statement in the “Right to Warn” proposal: “AI companies have strong financial incentives to avoid effective oversight, and we do not believe that corporate governance structures on their own are sufficient to change this.”
What if Bespoke doesn’t work?
Regulation is an obvious answer, and there is no question that more of it is needed. But it may not be enough by itself. Legislators often do not understand rapidly developing technologies well enough to regulate them with much sophistication. There is also the threat of regulatory capture.
This is why company insiders want the right to warn the public. They’ve got a front row seat to developing technology and they understand it better than anyone. With the freedom to talk about the risks they see, companies may be more motivated to take those risks seriously. This will benefit everyone, no matter what kind of AI risk keeps them up at night.
