Robotics and Biology Laboratory

Interactive Perception

Manipulating objects as dexterously as humans remains an open problem in robotics - not so much in carefully controlled environments such as factories, but in every day household environments, so-called unstructured environments. 

Interactive Perception, as the name suggests, is about acting to improve perception. The fundamental assumption is that the domains of perception and action cannot be separated, but form a complex which needs to be studied in its entirety. Using this approach, we try to design robot that explore their environment actively, in a way that reminds of how a baby explores a new toy.

Online Interactive Perception

Contact Persons

Manuel Baum

Aravind Battaje

Project Description

We developed an RGB-D-based online algorithm for the interactive perception of articulated objects. In contrast to existing solutions to this problem, the online-nature of the algorithm permits perception during the interactions and addresses a number of shortcomings of existing methods. Our algorithm consists of three interconnected recursive estimation loops. The interplay of these loops is the key to the robustness of our proposed approach. The robustness stems from the feedback from our algorithm which can be used to adapt the robot's behavior.

Active Outcome Recognition

Contact Persons

Manuel Baum

Aravind Battaje

Project Description

Robotic behavior can not only reveal properties of the environment, like kinematic degrees of freedom, but it can also be adapted to better understand the interaction between robot and environment itself. Robots need to assess the outcomes of their own actions and the sensory data created from task-directed behavior may not be sufficient for such an estimation. We work on adapting robotic behavior such that it is easier to estimate the outcomes of actions. This is a problem of learning interactive perception.

Generating Task-directed Interactive Perception Behavior

Contact Persons

Manuel Baum

Aravind Battaje

Project Description

Robots should not only passively process sensor information and then act based on the knowledge they extract from that sensor stream. They should (inter)actively shape that sensor stream by adapting their behavior so that the sensor stream becomes maximally informative for solving their tasks. We research how to generate such behavior in an engineering way, but also by learning it. In this research, we are interested in the relation between task-solving behavior, task-directed exploration and information based exploration. It's important for a robot to focus its exploration efforts on task-relevant variables, as pure information based exploration would generally not focus the robot's limited resources (due to its embodiment) enough for its behavior to be efficient.

 

Acquiring Kinematic Background Knowledge with Relational Reinforcement Learning

Contact Persons

Manuel Baum

Aravind Battaje

Project Description

If a robot faces a novel, unseen object, it must first acquire information about the object’s kinematic structure by interacting with it. But there is an infinite number of possible ways to interact with an object. The robot therefore needs kinematic background knowledge: knowledge about the regularities that hint at the kinematic structure.

We developed a method for the efficient extraction of kinematic background knowledge from interactions with the world. We use relational model-based reinforcement learning, an approach that combines concepts from first-order logic (a relational representation) and reinforcement learning. Relational representations allow the robot to conceptualize the world as object parts and their relationship, and reinforcement learning enables it to learn from the experience it collects by interacting with the world. Using this approach, the robot is able to collect experiences and extract kinematic background knowledge that generalizes to previously unseen objects.

 

Funding

Alexander von Humboldt

Science of Intelligence

Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy - EXC 2002/1 "Science of Intelligence" - project number 390523135

Publications

2023

Mengers, Vito; Battaje, Aravind; Baum, Manuel; Brock, Oliver
Combining Motion and Appearance for Robust Probabilistic Object Segmentation in Real Time
2023 IEEE International Conference on Robotics and Automation (ICRA), Page 683--689
IEEE
2023

2019

Martín-Martín, Roberto; Brock, Oliver
Coupled recursive estimation for online interactive perception of articulated objects
The International Journal of Robotics Research, Volume 41 (Issue 8) :741 - 777
May 2019
ISSN: 0278-3649
Martín-Martín, Roberto; Eppner, Clemens; Brock, Oliver
The RBO dataset of articulated objects and interactions
The International Journal of Robotics Research, 39 (9) :1013-1019
April 2019

2018

Pall, Elod; Sieverling, Arne; Brock, Oliver
Contingent Contact-Based Motion Planning
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Page 6615–6621
October 2018
Eppner, Clemens; Martín-Martín, Roberto Roberto; Brock, Oliver
Physics-Based Selection of Informative Actions for Interactive Perception
Proceedings of the IEEE International Conference on Robotics and Automation, Page 7427-7432
2018

2017

Pall, Elod; Sieverling, Arne; Brock, Oliver
Towards Motion Plans That React to Contact Events
RSS workshop: Revisiting Contact - Turning a problem into a solution
July 2017
Eppner, Clemens; Martín-Martín, Roberto; Brock, Oliver
Physics-Based Selection of Actions That Maximize Motion for Interactive Perception
RSS workshop: Revisiting Contact - Turning a problem into a solution
2017
Baum, Manuel; Bernstein, Matthew; Martín-Martín, Roberto; Höfer, Sebastian; Kulick, Johannes; Toussaint, Marc; Kacelnik, Alex; Brock, Oliver
Opening a Lockbox through Physical Exploration
Proceedings of the IEEE International Conference on Humanoid Robots (Humanoids)
2017
Sieverling, Arne; Eppner, Clemens; Wolff, Felix; Brock, Oliver
Interleaving Motion in Contact and in Free Space for Planning Under Uncertainty
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Page 4011-4017
2017
Sieverling, Arne; Eppner, Clemens; Brock, Oliver
Exploiting Contact for Efficient Motion Planning Under Uncertainty
RSS workshop: Revisiting Contact - Turning a problem into a solution
2017
Martín-Martín, Roberto; Brock, Oliver
Cross-Modal Interpretation of Multi-Modal Sensor Streams in Interactive Perception Based on Coupled Recursion
In IEEE, Editor, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Page 3289-3295
IEEE
In IEEE, Editor
2017
Martín-Martín, Roberto; Brock, Oliver
Building Kinematic and Dynamic Models of Articulated Objects with Multi-Modal Interactive Perception
In AAAI, Editor, AAAI Symposium on Interactive Multi-Sensory Object Perception for Embodied Agents
In AAAI, Editor
2017
Baum, Manuel; Brock, Oliver
Achieving Robustness by Optimizing Failure Behavior
Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Page 5806-5811
2017

2016

Martín-Martín, Roberto; Sieverling, Arne; Brock, Oliver
Estimating the Relation of Perception and Action During Interaction
International Workshop on Robotics in the 21st century: Challenges and Promises,
September 2016
Martín-Martín, Roberto; Höfer, Sebastian; Brock, Oliver
An Integrated Approach to Visual Perception of Articulated Objects
Proceedings of the IEEE International Conference on Robotics and Automation, Page 5091 - 5097
May 2016

2015

Buckmann, Marcus; Gaschler, Robert; Höfer, Sebastian; Loeben, Dennis; Frensch, Peter A.; Brock, Oliver
Learning to Explore the Structure of Kinematic Objects in a Virtual Environment
Frontiers in Psychology, 6 (374)
April 2015

2014

Höfer, Sebastian; Lang, Tobias; Brock, Oliver
Extracting Kinematic Background Knowledge from Interactions Using Task-Sensitive Relational Learning
Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Page 4342-4347
2014
Martín-Martín, Roberto; Brock, Oliver
Online Interactive Perception of Articulated Objects with Multi-Level Recursive Estimation Based on Task-Specific Priors
Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, Page 2494-2501
2014

2013

Höfer, Sebastian; Brock, Oliver
Learning Compact Relational Models for the Exploration of Articulated Objects
Proceedings of the ICRA Mobile Manipulation Workshop on Interactive Perception (ICRA),
May 2013