Elena Lacey | Getty Images
In October 2019, Idaho proposed changing its Medicaid program. The state needed the approval of the federal government, which requested public feedback through Medicaid.gov.
Around 1,000 comments have arrived. But half did not come from affected citizens or even internet trolls. They were created by artificial intelligence. And one study found that people couldn't tell the real comments from the wrong ones.
The project was the work of Max Weiss, a tech-savvy medical student at Harvard, but received little attention at the time. Given the rapid advances in AI speech systems, some say the government and internet companies need to rethink how they gather and review feedback to protect themselves from deep text manipulation and other AI-powered disruptions.
"The ease with which a bot can generate and submit relevant text that embodies human language on government websites is surprising and very important to know," said Latanya Sweeney, a professor at Harvard's Kennedy School who helped Weiss ethically conduct the Experiments advised.
Sweeney said the problems extend well beyond government services, but it is imperative that public authorities find a solution. "AI can drown out the language of real people," she says. "Government websites need to change."
The Centers for Medicare and Medicaid Services say they have introduced new protections for the public comment system in response to Weiss' study, but they refuse to discuss details. Weiss said he was contacted by the US General Services Administration, which is developing a new version of the federal government's website for posting regulations and comments to better protect it from falsified comments.
Government systems were previously the target of automated influence campaigns. In 2017, researchers found that over a million comments submitted to the Federal Communications Commission about plans to roll back the net neutrality rules were automatically generated, with certain sentences copied and pasted into various messages.
"I was a bit shocked to see that nothing more than a submit button stood in the way of your comment, which became part of the public record."
Weiss' project poses a more serious threat. In the past few years there has been remarkable progress in applying AI to language. When powerful machine learning algorithms are fed massive amounts of training data – in the form of books and texts taken from the web – they can create programs that can generate compelling text. Along with countless useful uses, the prospect is that all kinds of Internet messages, comments, and posts can be forged easily and less noticeably.
"When technology gets better," says Sweeney, "human language locations will be manipulated without humans knowing that it happened." Weiss was working for a health care consumer organization in the summer of 2019 when he learned about the public feedback process required to make Medicaid changes. Knowing that these public comments had influenced previous efforts to change government Medicaid programs, Weiss looked for tools that could generate comments automatically.
"I was a bit shocked to see that nothing more than a submit button got in the way of your comment, which became part of the public record," he says.
Weiss discovered GPT-2, a program released earlier this year by OpenAI, an AI company in San Francisco, and realized that it could generate false comments to simulate a fundamental wave of public opinion. "I was also shocked at how easy it was to tweak GPT-2 to actually spit out the comments," says Weiss. "It's relatively worrying in many ways."
In addition to the tool for generating comments, Weiss has developed software for automatically sending comments. He also conducted an experiment asking volunteers to differentiate between the AI-generated and human-written comments. The volunteers didn't do anything better than random guessing.
After posting the comments, Weiss notified the Centers for Medicare and Medicaid Services. He had added a few characters to make it easy to identify any fake comment. Even so, he says, the AI feedback stayed online for several months.
OpenAI released a more powerful version of its text generation program called GPT-3 last June. So far it has only been made available to a few AI researchers and companies, with some people creating useful applications like programs that generate email messages from bullet points. When GPT-3 was released, OpenAI said in a research report that it saw no evidence that GPT-2 was being used maliciously, despite being aware of Weiss' research.
OpenAI and other researchers have released some tools that can be used to identify AI-generated text. These use similar AI algorithms to identify telltale characters in text. It's not clear if anyone is using these to protect online commenting platforms. Facebook declined to say whether it uses such tools; Google and Twitter did not respond to requests for comment.
It is also not clear whether sophisticated AI tools are still being used to create fake content. In August, Google researchers released details of an experiment that analyzed over 500 million web pages using counterfeit detection tools. They found that the tools were able to identify pages that were hosting auto-generated text and spam. However, it wasn't clear if the content was created using an AI tool like GPT-2.
Renée DiResta, research manager at Stanford Internet Observatory tracking online abuse, expects more government sites to be attacked by fake texts. "Every time you have new technology, it's a tool in the hands of some and a weapon in the hands of others," she says.
Politically motivated misinformation has become a critical issue in American politics. Joan Donovan, research director for the Shorenstein Center for Media, Politics and Public Order at Harvard Kennedy School, warns that sophisticated AI may not be required to undermine people's sense of what is true. "People's feelings are frayed, and that makes them very susceptible to easy explanations rather than difficult truths," says Donovan.
This story originally appeared on wired.com.