Demand for data scientists has skyrocketed as many organizations struggle to make sense of – and reap value from – the unprecedented volume of data they have access to. Data analytics empowers businesses with actionable insights, enabling them to make better decisions, improve operational efficiency, understand customers, stay competitive, manage risks, forecast trends, and enhance marketing and sales effectiveness. It has become an invaluable tool for driving business growth and success in today’s data-driven world.
However, many of the skills and activities performed by data scientists are being commoditized – easily replicated or substituted. Most companies can leverage analytics with basic data knowledge combined with the appropriate software tools. Consequently, they no longer need to compete for this elusive talent.
Instead, they can shift their recruitment efforts to a much smaller pool of technical experts, combined with hiring or upskilling existing employees and using software tools to crunch their data. This can help them to significantly alleviate their shortage of analytics expertise.
About a decade ago, Harvard Business Review declared the role of data scientist to be the sexiest job of the 21st century. These professionals create automated actions or insights from typically large volumes of data. And they are still in high demand today.
According to the US Bureau of Labor Statistics, employment of data scientists is projected to grow 36% between 2021 to 2031, much faster than the average for all occupations. Companies are prepared to pay handsomely for this talent: the US median wage for a data scientist is $103,500, compared with $97,430 for overall IT occupations, according to the Bureau.
However, organizations need not fork out on human capital anymore: easy-to-use software is democratizing even those complex tasks that previously required data scientists. Artificial intelligence tools like DataRobot and AutoML help build models without the need to code. There are solutions to convert speech to text or detect objects, made available free-of-charge by open-source pioneers such as HuggingFace. This is great news for organizations that want to put data at the core of their business model.
For most data scientists, a considerable chunk of the work involves “scrubbing” the data: collecting, cleaning, structuring, and processing it to avoid creating fatally flawed algorithms. But in the next few years, we expect that menial data preparation will be carried out using software tools, such as IBM InfoSphere Information Server or innovative solutions from newcomers like Sweephy, a low-code data cleaning startup. Gartner predicts that 70% of new applications developed by organizations will use low- or no-code solutions by 2025, up from under 25% in 2020.
Even the tasks where data scientists once stood out due to their creativity, analytical strength, and business acumen are slowly being automated. That includes the creation of algorithms, which transform clean and reliable data into predictions and recommendations. The rapid growth and adaptation of generative AI tools, such as ChatGPT, has given even more impetus to this automation. Specifically, three forces are slowly eroding the need for these individuals.
First, we are starting to see large-scale commoditization of frequently used data services such as recommender engines, chatbots, and matching systems. The argument for building a customized data service is quickly losing ground in the face of emerging libraries of affordable and off-the-shelf algorithms and low-code solutions.
Second, the sheer volume of data, rather than the algorithm itself, is what is most valuable to organizations, especially those using advanced machine-learning techniques. The importance of volume over sophistication is visible in many winning applications, including TikTok, Spotify, and Netflix, which all leverage simple recommendation engines to generate value for users.
Third, many data scientists are technical specialists who lack a business background. This can lead to a lack of appreciation of the actual problems their tool is meant to address, as well as a misunderstanding of how it will be used by customers.
Taken together, these three forces are leading to a different role for the data scientist: from that of an astronaut (using state-of-the-art technology to venture into unchartered territory) to that of a champion race-car driver (using standardized technologies for real-world navigation).
A growing responsibility of the data scientist 2.0 is to bridge the gap between business and technology by translating broad business questions into clear data opportunities, as well as selecting the right set of services and tools to grasp the opportunities.
This implies different skill sets are needed, with a greater focus on adaptability, creativity, and business acumen. Organizations no longer need data scientists with PhDs in math or statistics to get clean data and smart algorithms, at least not to the extent they once did.
That means it has never been easier for companies to compete in today’s data-driven world. But there are some important considerations, such as starting early and seriously factoring investment in data into one’s business model.
We have three further recommendations for companies that want to harness the power of data analytics.
When it comes to building data capabilities, start by upping your average employee’s data skills rather than narrowly focusing on recruiting hard-to-find data scientists and then training them to understand your business. This will automatically reduce the number of data scientists you need.
Given the commoditization of data services and automation of tasks, it is becoming easier and more effective to teach functional experts the basics of adopting data science to address specific problems. In other words, data science is becoming easier to understand and deploy for the data-savvy businessperson. For instance, a Japanese consumer electronics giant has now made it compulsory for all new hires in the organization to learn how to use a basic data analytics tool, irrespective of function and level.
So, train suitable employees to act as “data translators,” who can bolster the essential bridge between technology and business that most organizations lack. It will also help you make the right trade-offs between off-the-shelf solutions and customized development.
However, this doesn’t mean you don’t need data scientists anymore — you do, but they should focus more on bringing technically sophisticated applications, scalability, and innovation to bear.
A growing responsibility of the data scientist 2.0 is to bridge the gap between business and technology by translating broad business questions into clear data opportunities, as well as selecting the right set of services and tools to grasp the opportunities.
As a large Dutch bank discovered, highly-skilled -technical roles are best organized as part of a separate team that can be tapped by different departments as a shared resource. The bank created a system in which the data team connects with the business through data translators, who act as data product owners. This ensures the data scientists stay focused, and work on projects that create real business value.
Imagine you are a consumer goods company that wants to increase sales by cross-selling to customers. You will need granular data to understand what they are buying and what their interests are, across your various products. This needs to be combined with simple models that provide individual-level recommendations. But far too often, we see companies embarking on an elaborate “data crusade” to collect consumer data, without having defined up front what they will use it for and how it will help deliver business results.
So, invest in systems that allow for the modularization of the most commonly used analytic tools and the data needed to drive them.
Ultimately, every company can benefit from the application of analytics to solve pressing problems. However, not all of them need a cadre of cutting-edge scientists to do so. As data analytics is democratized, organizations should rethink the mix of skills and capabilities they need to successfully navigate their business through the digital era.
The CDO, formerly a figurehead for regulatory compliance, is now a key driver of company strategy. Luca Condosta describes the essential skills that CDOs require to turn DE&I concepts into actual commercial...
The pandemic sparked a shift to remote work that’s proving hard to reverse. According to Prof. Manju Ahuja, hybrid models are the new norm—driven by employee expectations, technological advances, and a changing...
Use this diagnostic tool and roadmap to rebuild trust and confidence within dysfunctional teams – the first crucial steps toward a future of high performance....
Audio available
What to do when those in positions of authority behave in ways that contradict widely accepted norms of civility, empathy, and ethical leadership....
Audio availableExplore first person business intelligence from top minds curated for a global executive audience