Natural language research announcements

[Yonatan Bisk <bisk1 AT illinois.edu>]

-- If you have not done so already, please _email_ me ( bisk1 AT illinois.edu ) to meet with him this Friday --

When: Nov 19, Friday, 2pm.

Where: 3405 SC.

Speaker: Prof. Jeff Siskind (https://engineering.purdue.edu/~qobi/)

Title: Investigating Embodied Intelligence via Assembly Imitation and Learning to Play
Board Games

Abstract:
My research group is engaged in a concerted effort to ground the semantics
of natural language in computer vision and robotic manipulation.  We have designed a
novel custom robotic platform to support this effort and built three copies of this
platform to allow investigation of linguistic communication between robotic and human
agents.  We do this in the context of two tasks.  In the first, one robot builds an
assembly out of Lincoln Logs while a second robot observes that activity and
communicates those observations, in natural language, to a third robot who must
replicate that assembly.  In the second, two robots play a board game, while a third
robot---that does not know the game rules---observes the play and must infer the
game rules.  These tasks are specifically designed to support investigation into
integrating vision, robotics, natural language, learning, and planning with common
semantic representations and stochastic inference mechanisms.  This allows filling in
missing information from multiple modalities.  When the vision system cannot fully
determine the Lincoln Log assembly structure due to occlusion, it can ask question in
natural language or integrate information from multiple views with different camera
poses or taken at different assembly stages.  Likewise, when there are insufficient
training examples to learn game rules, the learner can ask questions that can be
answered either linguistically or by robotic demonstration.  I will discuss the common
stochastic inference mechanism built on top of a novel probabilistic programming
language augmented with automatic differentiation to support maximum-likelihood
estimation of rich complex models and how this software architecture together with
our hardware platform and rich integrated tasks reflect our vision for investigating
embodied intelligence.

Joint work with Andrei Barbu, Seongwoon Ko, Siddharth Narayanaswamy, and Brian
Thomas.

Bio:
Jeffrey M. Siskind received the B.A. degree in computer science from the Technion,
Israel Institute of Technology, Haifa, in 1979, the S.M. degree in computer science
from the Massachusetts Institute of Technology (M.I.T.), Cambridge, in 1989, and the
Ph.D. degree in computer science from M.I.T. in 1992.  He did a postdoctoral
fellowship at the University of Pennsylvania Institute for Research in Cognitive Science
from 1992 to 1993.  He was an assistant professor at the University of Toronto
Department of Computer Science from 1993 to 1995, a senior lecturer at the Technion
Department of Electrical Engineering in 1996, a visiting assistant professor at the
University of Vermont Department of Computer Science and Electrical Engineering
from 1996 to 1997, and a research scientist at NEC Research Institute, Inc. from 1997
to 2001.  He joined the Purdue University School of Electrical and Computer
Engineering in 2002 where he is currently an associate professor.  His research
interests include machine vision, artificial intelligence, cognitive science,
computational linguistics, child language acquisition, and programming languages and
compilers.

Thank you,

- Yonatan -