How would you teach a robot to make breakfast just the way you like it, wash your car, or give your children a French lesson? As any AI developer will explain, you simply need to show them what to do. Autonomous systems, including robots, can use large behavior models (LBMs) to understand and simulate human actions and behaviors. What’s more, they can repeat them indefinitely without getting bored or fatigued. On the contrary, the more they use them, the more creative they will get.
LBMs are related to, but distinct from, the large language models (LLMs) that power generative AI (GenAI). Data scientists train LLMs on huge volumes of data so that they can respond conversationally in natural language. LBMs, by contrast, learn from the behaviors of others as captured on camera and by sensors, as well as learning empirically, from their own actions.
At the Toyota Research Institute (TRI) in Massachusetts, US, one of the leading centers of LBM development, engineers are building robots capable of learning hundreds of separate, intricate skills using visual and tactile feedback systems. Once a robot has developed an extensive LBM skillset, it can reconfigure those skills to generate new behaviors, from selecting components for a production line to picking complementary ingredients from your larder to make your dinner.
Robots would need to incorporate an LLM capable of receiving, understanding, and responding to natural language queries from a human.
Many developers build LBMs using ‘diffusion policy,’ a process of AI training that involves breaking down any given action into much smaller movements and steps. The robot repeats these tiny actions in different contexts and environments until it successfully completes the assigned task in each of them. TRI, for example, has in a couple of hours trained a robot to load a dishwasher, while traditional coding might take a year to accomplish this.
Researchers at TRI have described their training as “kindergarten for robots.” The phrase hints at the potential such models have for future development. The robot can quickly learn a single action and then practice it to the point where it can perform the task in a range of conditions. Once mastered, the programming can then be instantly exported to a fleet of robots.
The potential is clear: LBMs are capable of driving natural, contextually aware behaviors that blur the line between human and machine, with robots taking initial cues from human supervisors and co-workers before developing and refining the actions according to their programming.
To make this scenario a reality will require technology that incorporates both LLMs and LBMs, with the two types of models working in tandem. Robots would need to incorporate an LLM capable of receiving, understanding, and responding to natural language queries from a human. The LBM would then action those queries, prompting the robot to follow the instructions.
“Eventually, we could see such robots in every workplace, augmenting or even replacing human operatives.”
For now, though, we should think in terms of more modest applications. Initially, industry is likely to implement LBM-powered robots in contexts where automation is already embedded in a predictable, controllable way, with a limited range of repeatable actions, such as automotive manufacturing.
But eventually, we could see such robots in every workplace, augmenting or even replacing human operatives. Researchers are particularly enthusiastic about applications in healthcare, entertainment, and leisure.
In the home, LBMs could power technology that replaces the current generation of personal assistants, offering an equally nuanced and sophisticated verbal service, combined with physical assistance.
Amid such excitement, however, caution is called for. It isn’t only science fiction fans who worry about a dystopian future in which robots start to take over. We need to put in place ethical and practical safeguards to mitigate the risks of LBM-powered technologies.
Underlining this, academics in the US published in 2024 a paper that warns of errors in the models prompting dangerous actions, or even cybersecurity weaknesses allowing malicious actors to manipulate the robots. It goes on to suggest that the robotics industry should conduct further safety research before building hardware that incorporates such models.
Many of the concerns that have dogged LLMs – those around opacity, bias, and hallucinations, for example – will also apply to LBMs, potentially with even graver consequences.
New government regulation may soon be on the horizon to control these threats, which, as in other areas of AI, will place the onus on policymakers to balance allowing innovators the freedom to pursue ambitious new goals with protecting workforces and the public.
With LBMs already beginning to move out of the lab, companies should develop and implement safety frameworks as soon as possible. For example, there is already speculation about integrating LBMs into Nvidia’s Hover controller for managing humanoid robots. Similarly, many believe Tesla’s Optimus project is also likely to incorporate LBMs, supporting the predictions by CEO Elon Musk that “humanoid robots [can] be the biggest product ever in history, by far”.
LBMs have significant advantages over LLMs in terms of predicting human actions and responses, incorporating contextual information, and interpreting emotional nuance.
Scientists have pointed out that LLMs have limitations in terms of understanding user actions and reactions and of how changes in environment could affect these. This, they suggest, is where LBMs could become effective beyond the physical realm.
In sales and marketing, for example, an LBM that predicts customer behavior throughout the purchasing journey could facilitate businesses in driving engagement and conversion. In healthcare, technologies that do a better job of predicting when patients are about to deviate from their care plans – skipping medication, say, or reverting to damaging behavior – could allow health professionals to make preventive interventions. In financial services, LBMs could be effective in identifying behaviors indicative of fraud.
LBMs have significant advantages over LLMs in terms of predicting human actions and responses, incorporating contextual information, and interpreting emotional nuance. These skills can power digital outputs in complex systems and environments, just as robotics engineers hope they will power physical outputs.
This year has begun with fascinating developments in the LLM world, with the emergence of China’s DeepSeek prompting questions about how easily such models can be built and replicated. But the evolution of LBMs promises to be an even bigger story. The robots really are coming, so watch this space.
All views expressed herein are those of the authors and have been specifically developed and published in accordance with the principles of academic freedom. As such, such views are not necessarily held or endorsed by TONOMUS or its affiliates.
As AI transforms industries, preparing young people for the future requires more than just mastering tools like ChatGPT. Success lies in developing deep knowledge, critical thinking, and the vision to understand how...
Audio available
A good AI use case results from a ‘matching exercise’ where value is found at the intersection of data sets and business problems and opportunities. This can be hard to achieve, but...
Audio available
AI carries the threat of bringing to life the chilling visions of Orwell and Huxley. We must work hard to fight against the convergence of micro-surveillance and digital inertia....
Audio available
Digital technologies, and artificial intelligence in particular, allow us to extract insights from data. This will allow industrial companies, the backbone of our economy, to be more resource-efficient, more productive, and more...
Explore first person business intelligence from top minds curated for a global executive audience