Humans have a way of understanding the goals, desires, and beliefs of others, an important skill that allows them to predict people’s actions. Are you taking bread out of the toaster? You will need a plate. Clear the leaves? I will grab the green bin.
This skill, often called “theory of mind,” is easily acquired by us as humans, but still difficult for robots. But if robots are to become truly collaborative assistants in manufacturing and everyday life, they must learn the same abilities.
In a new paper from the ACM/IEEE International Conference on Human-Robot Interaction (HRI) Awards Finalist, USC Viterbi computer science researchers aim to teach robots how to predict human preferences in assembly tasks so they can one day help. on everything from building a satellite to setting a table.
“When interacting with humans, the robot has to constantly guess what the human is going to do,” said lead author Heramb Nemlekar, a USC computer science graduate student working under Stefanos Nicolaidis, assistant professor of computer science. “For example, if the robot thinks the human needs a screwdriver to assemble the next part, it can get the screwdriver ahead of time so the human doesn’t have to wait. In this way, the robot can help humans complete assembly much faster. “.
But, as anyone who’s made furniture with a partner can attest, it’s hard to predict what one will do; different people prefer to build the same product in different ways. While some people want to start with the hardest parts to complete them, others may start with the easiest parts to save energy.
Making predictions
Most current techniques require people to show the robot how they want the assembly done, but that takes time and effort and can defeat the purpose, Nemlekar said. “Imagine having to assemble an entire airplane to teach a robot your preferences,” he said.
In this new study, however, the researchers found similarities in how an individual collects different products. For example, if you start with the hardest part when building an Ikea sofa, you will probably use the same delicacy when assembling a baby crib.
So instead of “showing” the robot their preferences in a complex task, they created a small assembly task (called a “canonical” task) that humans can complete easily and quickly. In this case, connecting parts of a simple model airplane, such as wings, tail and propeller.
The robot “watched” the human perform the task using a camera positioned directly above the assembly area, looking down. To detect human-operated parts, the system used AprilTags, similar to QR codes, attached to the parts.
The system then used machine learning to learn people’s preferences based on the sequence of their actions in a canonical task.
“Based on how a person performs in a small assembly, the robot predicts what that person will do in a larger assembly,” Nemlekar said. “For example, if a robot sees that a person likes to start with the easiest part of a small assembly, it will predict that they will also start with the easiest part of a large assembly.”
Building trust
In the researchers’ user study, their system was able to predict people’s actions with about 82% accuracy.
“We hope that our research can make it easier for people to show robots what they prefer,” Nemlekar said. “By helping each person in the way they prefer, robots can reduce their work, save time and even build trust with them.”
For example, imagine that you are assembling furniture at home, but you are not particularly comfortable and struggle with the task. A robot trained to anticipate your preferences can provide you with the tools and parts you need ahead of time, making the assembly process easier.
This technology can also be useful in industrial environments where workers are tasked with assembling products on a mass scale, saving time and reducing the risk of injury or accidents. In addition, it can help people with disabilities or limited mobility to assemble products more easily and maintain independence.
Quick learning preferences
The goal isn’t to replace people on the factory floor, researchers say. Instead, they hope this research will lead to significant improvements in the safety and productivity of assembly workers in human-robot hybrid factories. “Robots can perform non-value-added or ergonomically difficult tasks that are currently performed by workers.
As for next steps, the researchers plan to develop a method that automatically designs canonical tasks for different types of assembly tasks. They also aim to assess the utility of learning a person’s preferences from short tasks and predicting their actions in a complex task in different contexts, such as personal assistance in the home.
“While we observed that human preferences in montage production transfer from canonical to real-world tasks, I expect similar findings in other applications as well,” Nicolaidis said. “A robot that can quickly learn our preferences can help us prepare meals, rearrange furniture or carry out home repairs, which will have a significant impact on our daily lives.”