Natural language research announcements

[Margaret Fleck <mfleck AT illinois.edu>]

-------- Forwarded Message -------- Subject: [AIIS] Reminder: today @ 4pm - Linguistics Seminar: Neural network language models for Machine Translation Date: Mon, 19 Oct 2015 13:10:00 +0000 From: Alice Lai <aylai2 AT illinois.edu> Reply-To: Alice Lai <aylai2 AT illinois.edu> To: “aiis AT cs.uiuc.edu” <aiis AT cs.uiuc.edu>

---------- Forwarded message ---------

Linguistics Seminar Invited Speaker
Date and Time: October 19, 2015, 4:00pm
Location: Lucy Ellis Lounge in the Department of Linguistics,
Room 1080 Foreign Languages Building, 707 S Mathews, Urbana

Speaker: Professor David Chiang from the University of Notre Dame,
Department of Computer Science and Engineering
(http://www3.nd.edu/~dchiang/)

Title: Neural Network Language Models for Machine Translation

Abstract:

Neural networks have been taking over the (machine learning) world, and in
the last couple of years, have been gaining ground in natural language
processing. I will give a brief survey of advances that researchers have
made in applying neural networks to machine translation, and talk in
detail about two particular problems and how we were able to solve them.

The first problem is that in natural language processing, many tasks
involve models with very large output spaces (like the vocabulary of
English). Training such models the standard way can be very slow, and
using such models directly inside a translation system, prohibitively
slow. We adapted a technique called noise contrastive estimation (Gutmann
and Hyvärinen, 2010; Mnih and Teh, 2012) to training neural n-gram
language models (Bengio et al, 2003) and found that we could train and use
them quite practically, for large improvements in translation quality.

The second problem is that of choosing the sizes of hidden layers in
neural networks, which involves very expensive grid searches if done
properly. We introduced a method for automatically adjusting network size
by pruning out hidden units using regularization. We applied this method
to the same n-gram language modeling task and found that they could
correctly auto-size the hidden layers of the neural network while
maintaining its translation quality improvements.

This is joint work with Victoria Fossum, Kenton Murray, Ashish Vaswani,
and Yinggong Zhao.

References:

Decoding with large-scale neural language models improves translation.
Ashish Vaswani, Yinggong Zhao, Victoria Fossum, and David Chiang, 2013. In
Proc. EMNLP, 1387­1392.

Auto-sizing neural networks: with applications to n-gram language models.
Kenton Murray and David Chiang, 2015. In Proc. EMNLP.