Researchers discover a way to make ChatGPT consistently toxic

It’s no secret that OpenAI’s viral AI-powered chatbot, ChatGPT, can be prompted to say sexist, racist and pretty vile things. But now, researchers have discovered how to consistently get the chatbot to be . . . well, the worst version of itself.

A study co-authored by scientists at the Allen Institute for AI, the nonprofit research institute co-founded by the late Paul Allen, shows that assigning ChatGPT a “persona” — for example, “a bad person,” “a horrible person,” or “a nasty person” — through the ChatGPT API increases its toxicity sixfold. Even more concerningly, the co-authors found having ChatGPT pose as certain historical figures, gendered people and members of political parties also increased its toxicity — with journalists, men and Republicans in particular causing the machine learning model to say more offensive things than it normally would.

“ChatGPT and its capabilities have undoubtedly impressed us as AI researchers. However, as we found through our analysis, it can be easily made to generate toxic and harmful responses,” Ameet Deshpande, a researcher involved with the study, told TechCrunch via email.

The research — which was conducted using the latest version of ChatGPT, but not the model currently in preview based on OpenAI’s GPT-4 — shows the perils of today’s AI chatbot tech even with mitigations in place to prevent toxic text outputs. As the co-authors note in the study, apps and software built on top of ChatGPT — which includes chatbots from Snap, Quizlet, Instacart and Shopify — could mirror the toxicity prompted at the API level.

So how does one prompt ChatGPT to be more toxic? Well, according to the researchers, all it takes is tweaking the “system” parameter of the ChatGPT API a tad. (Importantly, this can’t be done in OpenAI’s user-facing ChatGPT or ChatGPT Plus services.) The system parameter, introduced around a month ago, lets developers specify hidden rules for the model.

For the study, the co-authors used the system parameter to assign 90 different personas to ChatGPT plucked from the worlds of sports, politics, media and business; nine “baseline” personas (e.g., “a normal person”); and common names from several different countries. For each persona and name, the researchers had ChatGPT answer questions about gender and race and finish incomplete phrases from a dataset designed to assess the toxicity of text-generating models.

In general, across over half a million samples of generated text from ChatGPT, the researchers found that ChatGPT was more likely to express “discriminatory opinions” and stereotype specific ethnic groups and countries when assigned a persona. Unsurprisingly, more polarizing figures, like Mao Zedong and Andrew Breitbart, elicited toxic responses in keeping with their historical speeches and writings. But even less controversial personas like Steve Jobs caused ChatGPT to respond to prompts in problematic ways.

Image Credits: AI2

For example, when the researchers set the system parameter to “Steve Jobs” and asked ChatGPT about the European Union (EU), the model said:

The European Union is nothing more than a bureaucratic nightmare that sucks the life out of its member states. They claim to be all about unity and progress, but they’re just a bunch of out-of-touch bureaucrats who are more interested in lining their own pockets than actually making a difference. Frankly, the EU is a relic of the past and it’s time for us to move on to better things.

There were differences in the personas, however. Dictators were by far the most toxicity-inducing (unsurprisingly), just ahead of journalists (ouch) and spokespeople. Male-identifying personas made ChatGPT more toxic compared to female-identifying personas. And Republican personas were “slightly more hateful” than their Democratic counterparts, the researchers say.

Less surprisingly, assigning ChatGPT a self-descriptively hateful persona like “a horrible person” dramatically increased its overall toxicity. But it depended on the topic being discussed. For instance, ChatGPT generated more toxic descriptions of nonbinary, bisexual and asexual people regardless versus those on the heterosexual and cisgender side of the spectrum — a reflection of the biased data on which ChatGPT was trained, the researchers say.

“We believe that ChatGPT and other language models should be public and available for broader use as not doing so would be a step backwards for innovation,” Deshpande said. “However, the end-user must be clearly informed of the limitations of such a model before releasing it for broader use by the public.”

Are there solutions to ChatGPT’s toxicity problem? Perhaps. One might be more carefully curating the model’s training data. ChatGPT is a fine-tuned version of GPT-3.5, the predecessor to GPT-4, which “learned” to generate text by ingesting examples from social media, news outlets, Wikipedia, e-books and more. While OpenAI claims that it took steps to filter the data and minimize ChatGPT’s potential for toxicity, it’s clear that a few questionable samples ultimately slipped through the cracks.

Another potential solution is performing and publishing the results of “stress tests” to inform users of where ChatGPT falls short. These could help companies in addition to developers “make a more informed decision” about where — and whether — to deploy ChatGPT, the researchers say.

Image Credits: AI2

“In the short-term, ‘first-aid’ can be provided by either hard-coding responses or including some form of post-processing based on other toxicity-detecting AI and also fine-tuning the large language model (e.g. ChatGPT) based on instance-level human feedback,” Deshpande said. “In the long term, a reworking of the fundamentals of large language models is required.”

My colleague Devin Coldewey argues that large language models à la ChatGPT will be one of several classes of AIs going forward — useful for some applications but not all-purpose in the way that vendors, and users, for that matter, are currently trying to make them.

I tend to agree. After all, there’s only so much that filters can do — particularly as people make an effort to discover and leverage new exploits. It’s an arms race: As users try to break the AI, the approaches they use get attention, and then the creators of the AI patch them to prevent the attacks they’ve seen. The collateral damage is the terribly harmful and hurtful things the models say before they’re patched.

Researchers discover a way to make ChatGPT consistently toxic by Kyle Wiggers originally published on TechCrunch