In this paper we propose a trainable system that learns grounded language models from examples with a minimum of user intervention and without feedback. We have focused on the acquisition of grounded meanings of spatial and adjective/noun terms. The system has been used to understand and subsequently to generate appropriate natural language descriptions of real objects and to engage in verbal interactions with a human partner. We have also addressed the problem of resolving eventual ambiguities arising during verbal interaction through an information theoretic approach.

This paper introduces “outside-in design” as a collaborative approach to social robot design and human-robot interaction research. As an interdisciplinary group of social and computer scientists, we follow an iterative practice of collecting and analyzing data from realworld interaction, designing appropriate robotic perception and control mechanisms, developing models of interaction through automatic coding of behaviors and evaluation by human subjects, and validating the models in embodied human-robot interaction. We apply this approach in the context of shadow puppeteering, a constrained interaction space which allows us to study the foundational elements of synchronous interaction and apply them to a robot. We contribute to both social and computer sciences by combining the study of human social interaction with the design of socially responsive robot control algorithms. Interaction with robotic technologies in the real world poses both social and technical challenges. For a robot to collaborate seamlessly with humans in an everyday activity, it has to be situationally aware, able to take advantage of the human’s knowledge of the world, and adapt its behavior accordingly. To enable a socially interactive robot to perceive and display relevant social behaviors, designers must solve complex problems in real-time perception and control involving multiple mechanical and computational systems. Designing robots for social interaction also calls for expertise in analyzing social behavior to understand the factors that make people respond to robots as social actors. The challenges of social human-robot interaction suggest that it is difficult to neatly ‘divide and conquer’ social robot design through partial solutions bounded off within social and computational disciplines. This paper describes a collaborative practice bringing together computational and social expertise in the exploration and design of social human-robot interaction. We use an “outside-in” 1 design strategy, iterating between real