Is artificial intelligence the answer to improving diversity, equity and inclusion?

However committed CEOs say they are about creating organisations that are diverse, equal and inclusive, the simple fact is that businesses are run by people and people are not – as it turns out – very good at suppressing bias.

Judd Kessler and Corinne Low, professors at Wharton School, University of Pennsylvania, tested this out when they asked hiring managers from prestigious firms to rate 40 fake CVs each. Featuring names indicative of race and gender, as well as differing experiences (such as internships and/or previous work), the recruiters knew the resumes were hypothetical but their responses were used to match them with real job seekers, providing an incentive for them to carefully rate the fake resumes.

So what did they find? Well, perhaps unsurprisingly, the results were conclusive: despite 90% of these firms outwardly declaring they positively factored in DEI in their hiring, their ratings suggested otherwise.

“No aggregate preferences were given for minority or female candidates,” the authors said. “Firms rated minority and female candidates significantly lower than white males; while prestigious internships on the CVs of white males saw their ratings get boosted by 50% more than the same internship on a female or minority resume.” Oh, and not forgetting that socio-economic bias was introduced too. “Internships that firms valued were typically unpaid, and therefore only possible for a privileged few,” they said.

Experiments like this help explain why virtually all DEI metrics have remained stubbornly difficult to improve. Even when attempts are made to improve HRDs’ biases, success is not always a given. One meta-analysis of 426 studies of anti-bias training concluded that its results were (at best) “weak”.

Against this background, it's no wonder there are those who believe advances in technology should now be left to deliver where humans have failed. In other words, artificial intelligence (AI) and machine learning should take over and remove humans from as much of the decision making as possible.

“There’s certainly a view that if humans alone are in charge, it’ll take 20 years for change to filter through, and so AI can certainly give firms a head start,” says Jose Rodriquez, chief data protection officer at Cornerstone. He adds: “DEI hasn’t improved, because HR keeps doing the same thing, so something new is needed.”

AI can even the playing field

For many the default position seems to be that AI is no better, as algorithms are themselves built by humans and so carry the same biases, as Amazon famously found out. “Only 15% of AI and machine learning is designed by women, so the people designing the rules and keywords are themselves not diverse,” argues Serge da Motta Veiga, professor of human resource management at EDHEC Business School. He specifically studied AI’s impact on more ethical hiring and concluded “it isn’t perfect”. Worse still, he argues, is altering AI’s calculations to positively discriminate. This is widely seen as wrong.

But others argue that both these elements are missing the point and that AI’s role, certainly in recruitment, is less about promoting certain groups but just taking things to a point where everyone has a chance. In other words, tipping towards a neutral position. As da Motta Veiga adds: “You only increase diversity by not eliminating people who might be suitable.

Brendan O’Hara, CTO and co-founder of organisational change consultancy Untapped AI, says leaders need to start by understanding what the purpose is of using AI in the business. “If used to make decisions it’s bad; but if it’s trying to inform and enable, then you almost certainly want bias in there anyway – as that’s what makes a company self-aware of the problems a company has.” He says: “Data lives, it evolves. It loses or gains bias over time, and so its use is as a way of extracting problems.”

This thinking ultimately brings AI back to its relationship to HR. “The real bias is not seeing people for who they are,” says Simon Robinson, co-author of Deep Tech & the Amplified Organisation. “It is now possible to tweak your deep learning and train your AI to overcome bias – in effect removing the machine bias from human bias. But where we actually see more use is in AI supporting staff throughout the whole employee experience – from chatbots to performance management and promotions not being overlooked based on age/gender/ethnicity.

Cornerstone’s Rodriquez adds: “Successful use of AI in DEI is about changing the mentalities of the people using it.”

Pearl Kasirye, founder and HR lead at Pearl Lemon Group, is one such person who believes a mix of AI and in-person interventions presents the best solution: “AI is an equaliser. It can help get diverse people through the door but it will not be there to make the company culture inclusive and culturally sensitive,” she says. “AI does not have intuition, cultural sensitivity or intercultural competence.”

She continues: “Let's say you change the AI to be more accepting of a diverse group of people. But once those people start working it's inclusive company policies that will retain them. So the two things need to go hand in hand. Employers can't just look at attracting diverse applicants and leave it at that.”

Using technology to track potential and visibility

Some of the best AI technology will track employees’ skills and competencies and predict career advancement versus their peers to give HR data on whether people are simply reaching their potential. Experts argue it’s this that can be transformative.

“It’s absolutely possible to remove the bias from AI,” says Max Blumberg, visiting researcher at Goldsmiths, University of London and founder of Blumberg Partnership. “But even if it’s not, it can come close, and it’s better than what some HRDs are doing already.”

AI does show promise when it comes to helping remove bias, agrees David Helfrich, client partner and head of the Americas at talent mapping and executive search firm Armstrong Craven. “It has the potential to be so right, whether it’s technology that assesses how job ads are written (the text itself may inadvertently put some protected groups off) or technology that looks at blind CVs and assesses just for skills and potential rather than age/sex or socio-economic/ethnic background.”

One of the newest providers in the market is Natter, which takes AI beyond recruitment, performance reviews, promotion and L&D to breaking hierarchical barriers and creating virtual ‘watercooler’ moments. The AI will suggest whom people should introduce themselves to in a company, based on their skills, interests, experiences and previous answers, creating much better visibility between people of different groups. CEO Charlie Woodward says: “Machine learning assesses the value of these conversations and the platform develops data around how conversations take shape and between whom.” He adds: “Every company faces the same problem of proximity bias but AI being able to improve people’s at-work experiences is where we see it hitting the DEI sweetspot.”

AI’s potential certainly isn’t going unnoticed. Some 89% of executives in a recent poll by Intel admitted that technology would make achieving DEI goals easier, while 35% said technology bolsters DEI and is critical to achieving it.

Certainly, some HRDs appear to be much more interested in its potential rather than its pitfalls. “For us, our thinking about AI and how it can impact DEI is still in its infancy, but nevertheless we’re looking right now at how we can use our own technology tools internally,” says Charlotte Forsyth, HR director at Streetbees, which provides consumer insight to marketers. “We’ve got the tools for real-time analysis of sentiment and so the opportunities are huge. What always struck me about technology is that you tend to over-estimate its pace of change but underestimate its impact, and I think that while AI will take a little longer to infiltrate companies’ DEI agendas, I think they should see the impact of its benefits quite quickly.”

“We’re hugely optimistic about AI’s role in DEI, but it has to be in the context of an expanded view of leadership,” says Robinson. ”We are at a turning point in terms of where technology can go but it’s not the only answer. It can’t solve in-built and deep-seated issues within organisations.”

Because things seem to be at an inflection point, Armstrong Craven’s Helfrich urges considered caution. The essence of success is still human thought, he says.

“Someone once told me that you don’t send in a droid to do a Jedi’s job. There are some tasks that AI can do effectively, just as a droid can do technical tasks well. But the capability for AI to assess emotional intelligence (EI) or cognitive IQ is still up for debate. AI can’t do the deeper processes and analyses that humans can do, thanks to that EI and cultural competency. Regardless of the great potential AI has, it cannot replace human functions.”

So is AI the way forward for helping move the diversity needle further? Many believe – once bias worries are settled – that it is.

Dr Joanna Abeyie, a diversity, equity and inclusion leader and trustee of The Lord Mayor’s Appeal, is sanguine. “It may not be transformational yet, but it beats what some HRDs are doing. It has its hurdles – and bias is one of them – but there are definitely benefits.”

Adds author Robinson: “AI can be a means to do things better – as long as it’s in tune with the cultural aspects of an organisation. Its best use is to find people. After that, though, normal HR needs to kick in – mentoring and having the right policies in place to ensure these people stay.”

Top tips on making AI a DEI help rather than hindrance

1. Stick to the rules
   New EU rules proposed in 2021 could shape how companies use AI data. AI systems that are used in  employment and worker management were put on the EU’s high-risk list. This category requires that any software that is used in a recruitment procedure will be subject to strict obligations before they can be put on the market.
    
2. Understand the different sets of biases
   These include affinity bias (when AI responds more positively to candidates who share certain characteristics, backgrounds and personality traits) and halo effects (too heavy a reliance on a single positive trait rather than interrogating skillsets).
    
3. Take a more holistic approach
   Strong DEI strategies should start by understanding the entirety of the business – beginning with ongoing data collection that looks not just at basic demographics but operational, experiential and performance metrics too.