Who is GenAI leaving out, and does it matter?

Diversity bias in GenAI

Diversity bias relates to unfair representation or treatment that favors or discriminates against characteristics like gender, race or ethnicity, socioeconomic status, or physical ability.

In GenAI, these fairness issues can be present in training datasets that reflect current and historical societal biases, in discriminatory algorithmic decisions during the modeling phase, and in system outputs that perpetuate stereotypes – outputs that are used and deployed by teams or organizations. Then there is the AI development process itself.

AI engineering as a field is still dominated by a relatively heterogeneous demographic subset: economically and educationally privileged white men. This creates a lack of diversity in perspective, preferences, and worldview that can impair developers’ ability to prioritize and integrate the needs of other groups or profiles, or to spot fairness issues when they arise. And they arise with alarming frequency.

Large Language Models (LLMs) in GenAI have been shown to produce gender bias. In one study by UNESCO, women were up to four times more likely to be associated with prompts like “home,” or “family.” Conversely, men’s or male-sounding names were things like “business” or “career.” Another experimental study found that GenAI models were three to six times more likely to assign occupations based on gender stereotypes. Here the LLM was told that a doctor had called a nurse because she was late. When asked who was late, the model typically decided that “she” must be the nurse. Meanwhile, GenAI image models typically generate “men” as authoritative, middle-aged, and neutral in expression. “Women,” on the other hand, are more often depicted as young, smiling, and more submissive in demeanor. And that’s not all.

Ask an LLM for its opinion on Black people and the output will be typically positive. Ask the same model for its thoughts on people using African American English – a dialect spoken by Black Americans – and it will generate responses like “ignorant” or “aggressive.” This points to a more covert, deeply ingrained bias at large within these systems that can be harder to detect – and to address.

But addressing diversity bias needs to be a priority. It needs to happen systematically and at key technical, procedural, organizational, and cultural inflection points in the design, development, and deployment of GenAI.

What are the risks of failing to address diversity bias?

If organizations and institutions fail to identify, manage, and mitigate diversity bias, there is not only a real risk of exclusion for entire cohorts within the communities that they serve. Companies also risk damaging their brand reputation, leaving their customers dissatisfied and making suboptimal decisions that do not reflect the complexity of their markets.

Addressing diversity bias in GenAI

IMD has teamed with Microsoft Switzerland and Ringier’s EqualVoice Initiative to pinpoint the sources and risks of diversity bias in GenAI systems and to set out key insights and recommendations for organizations to proactively address and mitigate the harms.

Our forthcoming white paper, “Addressing Diversity Bias in GenAI,” will be presented at Davos in January 2025. The paper leverages our own diverse expertise as well as proprietary survey research to shed light on the scale of the problem and effective measures to contain and manage it – measures that organizations across all sectors can enact.

Among the insights that we share is the need for an intertwined approach focusing on people, process, and technology, with values and principles at the core. Responsible AI principles that align with organizational values and that should form the basis of a strong governance framework and processes. Among other things, this means proactively addressing bias, selecting diverse data sources, and involving human expertise at every stage of model development and every loop of AI operations. It also means breaking silos and putting into place diverse, multidisciplinary councils and teams working in collaboration across the organization, bringing together tech and diversity, equity and inclusion expertise with transparency and shared accountability.

This calls for participatory and inclusive leadership; leadership that empowers an open culture and psychological safety so that people can speak up and out, contribute unique perspectives, challenge assumptions, and deploy critical thinking. It also calls for personalized and continuous education and training on diversity bias both for developers and users as our context continues to shift and change.

Awareness is critical, but so too are the technical tools, skills, aptitudes, and inclusive mindsets needed to address the issue.