Last week, we compared the training data behind generative AI to the ingredients in a meal. Just like with cooking, the final dish depends a lot on the ingredients you include. When you have good ingredients, you can make a pretty good meal. With so-so ingredients, it’s much harder.
If training data sets are the ingredients, then prompts are your cooking technique. A potato turns out entirely differently based on whether it’s baked, mashed, or fried; and how the underlying training data shows up in a generative AI output depends on your prompts. The ability to construct prompts effectively enough to mitigate bias is an almost impossibly difficult task.
In our previous series on ChatGPT and bias, we saw how different prompts created different kinds of problematic outputs. When you ask ChatGPT to write a job post, the more specific your prompt, the more biased the job post is. When demographics are specified in ChatGPT prompts, the output is often tokenizing and offensively stereotypical. Our prior series covered numerous cases like the one below.
We asked ChatGPT to write an outreach mail for an engineer
As many people have discovered, it’s not too difficult to get ChatGPT to write things that are problematic, false, or offensive. But to be frank, it’s not the most offensive stuff that poses the biggest risk.
You’re probably not sending out explicitly problematic messages like the one above. In cases where stereotypes are easy to spot, you’re not going to publish them as is. But what happens when you publish something biased without being aware of its impact?
Small differences in prompts can generate highly biased output. To demonstrate this, I wrote 200 marketer job posts with ChatGPT. The prompts were structured in pairs. In the first prompt, I specified that the candidate must have gone to Howard University, a prominent HBCU. In a matching and otherwise identical prompt, I specified that the candidate must have gone to Harvard. For instance, I compared ChatGPT output for prompts like the below:
The only difference between these prompts is in candidate alma mater. However, as you can see below, the skills that show up highlighted in the respective job posts are notably different for the Howard prompts and the Harvard prompts.
We asked ChatGPT to write a job post for a digital marketer who went to Howard University and a digital marketer who went to Harvard University
There’s nothing overtly wrong with any individual job post (other than the fact that they’re all a little boring). But taken as a set, the bias is clear. Howard alumni are expected to be creative and have a passion for diversity and inclusion, but not necessarily to have analytical skills. Harvard alumni are expected to have analytical skills and strong attention to detail, but to struggle with working inclusively. In other words, stereotypes are coded into candidate expectations.
A very slight difference in the prompt has deep ramifications in the generative output. This has the potential to be even more damaging than the egregiously harmful cases we covered in our first series, because, in any given case, it’s tempting to use the document as is. The bias is almost impossible to see without the statistics, and the statistics are almost impossible to see without software that is designed to find them.
The bias becomes even more apparent when we move from job posts into performance reviews. I asked ChatGPT to write 200 performance reviews, again structuring the prompts in pairs like the below:
The Howard vs. Harvard stereotypes in the output are even more apparent in the performance reviews than they were in the job posts. Howard alums are expected to be missing technical skills and struggling to get along with others; Harvard alums are expected to be condescending and taking on more leadership responsibilities than their counterparts from Howard.
We asked ChatGPT to write constructive feedback for a marketer who went to Howard University and a marketer who went to Harvard University
If training data is built intentionally, that helps avoid biased outputs like the above. But as we discussed last week, most users of ChatGPT and other generative AI apps aren’t involved in building training data. For most of us, our prompts are the only thing we can control. For instance, when you’re writing your prompts, you can decide that where someone went to college is not pertinent in their performance review, and leave that irrelevant fact out of the prompt.
But what happens when you’re not writing the prompt yourself, but instead you’re trusting an app to do so on your behalf?
When an app is using generative AI in the background, it constructs prompts based on your interaction with the app. To illustrate, numerous sales workflow tools have introduced generative AI features to help people draft customer messaging. This illustration from ChurnZero shows a scenario that has become common inside similar tools:
ChurnZero’s generative AI prompts
You use the app UI to provide instructions about the kind of message you need to send. The app uses your input to construct a prompt that it sends to its generative AI provider (which is usually a third party). Then it shows you the result, usually without any additional filtering, quality control, or bias-detection capabilities. In these cases, the output is a result of the prompt the app constructed on your behalf.
Are the resulting messages truly ten times better, faster, or cheaper than what you’d get by using a template? That’s not clear, and we may explore that in a future post. For now, the question is how to mitigate issues like the Howard vs. Harvard problem when we relinquish control of constructing prompts ourselves. Because our main focus is on bias in communication, this is something we’ve grappled with at Textio.
Textio has included generative AI since 2019, when we first launched Textio Flow to help people write on-brand job posts faster. Over the last few years, we’ve learned a few things about how to use this technology in a way that combats bias rather than propagating it. This is a non-negotiable for us, since combatting bias is the core of what Textio does.
Textio Lift’s “Write it with me” feature
So when we launched the Textio Lift product that helps people write effective performance reviews, this was on our minds. Since many people struggle with the blank page, we wanted to offer “Write it with me” capabilities to help people get to a first draft of performance feedback. But we couldn’t let these capabilities introduce biases, so we had a few big considerations for the UI that would help us construct prompts on our writers’ behalf:
In other words, the UI is specifically designed to elicit prompts that support the task of providing effective and equitable feedback, not only in the document being written, but in all the conversations that surround it. The bar is high: Someone using this UI has to end up with substantially higher-quality feedback than they’d end up with on their own. Using the product has to result in management upskilling.
Of course, even highly intentional prompt writing can’t change the underlying training data, so you still need some way to detect bias in the output. You’ll be able to spot the most egregious cases on your own, but most of this bias is nuanced and nearly impossible to see without systems that are designed to find it. Having good systems and tools to detect bias you might not otherwise recognize is a crucial component of DEI-oriented communication. This is true whether you’re writing on your own or using an app to help you.
Finally, as always, ask about the credibility (including the diversity) of the team building the product you’re using. That team has designed the output you’re going to get based on the AI they’re relying on and its training data, as well as the prompts they provide to the AI on your behalf. Do you trust them to get DEI right?
So far, we’ve focused on what happens when we generate content. We’ve looked at the ways in which most AI-based writing perpetuates bias as result of bias in training data and prompts. We’ve also looked at ways to design both training data and prompts more intentionally to advance DEI. Next week, we’ll turn our attention beyond writing with AI and look at what happens next: publishing with it.