Leadership thinker James Belasco famously remarked that coaching “is destined to be the leadership approach of the 21st century.” It’s perhaps no surprise that Harvard research suggests the manager of tomorrow will be less like an army commander and more like a basketball coach. Being able to coach effectively is not something that comes naturally, however, especially as many managers lack formal training.
Leading like a coach requires strong communication skills: asking good questions, resolving conflicts, and providing meaningful feedback. Yet many executives lack these abilities. Traditionally, the best way to develop them has been through human trainers who observe conversations and provide feedback. But given the capabilities of large language models, we wondered, could AI-powered tools become effective sparring partners for sharpening executive coaching skills?
We created a GPT-4-powered tool to provide feedback to executives and tested it with 167 global executives across industries. Our findings: AI-based coaching interventions can efficiently improve essential leadership capabilities.
AI tools can listen to conversations between an executive and someone else. Afterward, the executive asks the tool to assess their communication style. Specific questions yield better outcomes.
Executives can use AI tools in two ways. First, they can treat the AI as a leadership coach, asking for help in preparing for or analyzing difficult conversations. They can also prompt the tool to role-play different scenarios.
Consider an executive uncertain about following up with a team member. She writes a 250-word description of the situation – a moment of reflection that provides clarity. She then feeds this into the tool, prompting it to act as a coach with questions like “What would you advise?” or “How can I deal with this productively?” Specific prompts work best, such as “Which communication style would you recommend?” or “How do I apply a supportive style succinctly?”
Second, AI tools can listen to conversations between an executive and someone else. Afterward, the executive asks the tool to assess their communication style. Specific questions yield better outcomes. Asking the tool to provide examples delivers useful, if sometimes surprising, insights. Questions like “Was my word choice appropriate?” or “Were the dynamics productive?” tapped into AI’s ability to understand the context of the conversation. It was particularly useful when the AI based its responses on a specific coaching framework, as then executives could ask specifically about things like their choice of communication style in particular situations.
We used a version of GPT-4 fine-tuned with coaching conversations analyzed using John Heron’s framework. Heron categorizes interactions into six types: prescriptive (offering advice), informative (sharing knowledge), catalytic (encouraging self-discovery), cathartic (sounding out emotions), supportive (providing encouragement), and confronting (challenging assumptions).
Executives coached each other using real career situations while our tool listened. Afterward, the coach asked questions such as:
Human observers also watched the conversations to critically assess the AI’s feedback.
“Initially skeptical, I soon recognized the importance of understanding how to measure certain outcomes of our leadership. Technology can significantly enhance our leadership skills.”
The tool recognized Heron’s coaching styles precisely and provided appropriate feedback on their use. The AI-generated recommendations aligned with human observers’ findings.
One senior executive from a Scandinavian company said, “Initially skeptical, I soon recognized the importance of understanding how to measure certain outcomes of our leadership. Technology can significantly enhance our leadership skills.” A Japanese executive said that the experience showed how AI can do more than just crunch data, while a Moroccan executive marveled at how practical and interactive the approach was.
The executives found the assessments useful, and sometimes surprising. In one instance, an executive had considered another approach but decided against it. The tool detected this unexpressed choice and highlighted it as an alternative, which was an unexpected revelation.
The tool proved most helpful with specific prompts. “Can you give me specific examples?” yielded the most valuable feedback.
We created a framework measuring how “novel” and how “useful” the tool’s assessments proved. The first dimension measures the discrepancy between executives’ self-perception and the tool’s assessment. The second measures how useful participants found the feedback. The results fell into four distinct groups.
The most common was the “zone of learning,” in which feedback was both surprising and useful. It is pleasing that 55% of results fell into this category, as this is the most transformative. Participants said that the insights challenged their assumptions and opened up new ways of thinking. They appreciated the granular attention to detail the AI brought, including in areas like the tone of voice used during conversations.
The next most common category was the “zone of validation,” which is where the feedback from AI aligned with expectations and reinforced our existing knowledge. This can be comforting, but it also runs the risk of complacency, as it doesn’t invoke change.
The last two categories were fairly rare, with the “zone of irritation” occurring for just 10% of respondents. This occurred when people felt unhappy with the feedback, often because it jarred with their self-perception or because they felt it was irrelevant. The “zone of indifference” was felt even less, but a few participants nonetheless felt that feedback was too generic or obvious.
The technology can provide accurate and detailed feedback, but it’s nearly always better when it’s combined with human interpretation.
The experiment had some important limitations to consider as we draw wider lessons from the experience. The main consideration is that AI is only as good as the data it has, and there are inevitable question marks about the security and confidentiality of sensitive conversations. We need to ensure that consent is given and security protections are in place.
We also need to be aware of the potential for hallucinations or other forms of inaccurate feedback. It’s important that leaders are able to critically evaluate the feedback they receive and cross-reference it with peer feedback and their own personal observations. If you can ground your conversations in proven frameworks, like Heron’s, then these risks are reduced somewhat. These frameworks enable executives to identify instances where the tool hallucinates.
AI can also struggle with cultural nuances, emotional dynamics, and unspoken cues, and these are often critical in coaching. These gaps highlight the need to use AI tools as complements, not replacements, for human expertise.
The first lesson is that feedback is most effective when it’s given under structured frameworks and via specific and tailored prompts. Frameworks like Heron’s are a good starting point. We found that the feedback improved significantly when prompts were as specific as possible, as this allowed the conversation to be deeper and more structured.
Secondly, AI is likely to provide feedback that challenges our self-perceptions. Indeed, the most valuable feedback is often when AI’s perspective differs from our intuition. It’s then that uncomfortable observations are surfaced. We should resist dismissing these automatically and instead ask follow-up questions and experiment to see if the observations have merit.
It’s also important to combine perspectives rather than rely purely on AI. The technology can provide accurate and detailed feedback, but it’s nearly always better when it’s combined with human interpretation. This isn’t a case of human coaches being automated away, and there is still tremendous value in discussing the AI feedback with a human coach or mentor.
Last but not least, this should be viewed as an ongoing process that is continuously refined. AI is increasingly capable of retaining context and memory, so as you learn more about the right prompts and the best way to elicit the most effective feedback, you will also learn how to use the tool with increasing precision and confidence. In this way, AI becomes not just a tool, but a catalyst for sustained leadership development.
If you want to start using AI as a professional sounding board, here are three tips to help you get the most from the process:
Ecosystems promise a new way to compete, yet most squander the opportunity by bolting them onto old strategic logic. Mark Greeven makes the case for putting ecosystem thinking at the heart of...
Audio available
Senior leaders can leverage AI to boost creativity, guide decisions, and optimize talent, while balancing risks like bias and overreliance....
AI may become one of the most significant leadership opportunities for women in decades. Its impact will depend on how capability, governance, and leadership are built around it....
AI is eroding entry-level roles, threatening future leaders. Erik Brynjolfsson warns organizations to rethink hiring and invest in strategic early-career talent....
Explore first person business intelligence from top minds curated for a global executive audience