Proposal

Workshop URL: http://npbayes.wikidot.com

Topic

Nonparametric Bayesian models are a large class of probabilistic models with unbounded capacity. Overfitting is prevented by the Bayesian approach of integrating out all the parameters and latent variables using approximate inference techniques like Markov chain Monte Carlo and variational Bayes. As a result, the models will automatically infer the right amount of complexity needed to model a given data set.

Nonparametric Bayesian models were first studied in the statistics community, rediscovered in machine learning, and have since flourished, with a growing variety of models, and applications of such models in disciplines such as information retrieval, natural language processing, machine vision, computational biology, cognitive science and signal processing. Furthermore, nonparametric Bayesian models have also driven a number of efforts in developing new languages to describe probabilistic models, and more efficient and accurate approximate inference algorithms for graphical models.

Motivation

This is the third in a series of successful workshops organized by Teh et al. The first two were at NIPS in 2003 and in 2005. There has also been an ICML workshop in 2006. Since then there have been significant advances in the field, including many novel models, new approximate inference techniques, and compelling applications in a variety of disciplines. As a result, a larger community has grown around this topic. However, for this field to deliver on its promise of a flexible and easy to use probabilistic language for knowledge representation, a number of important issues must be addressed.

The first one is software. The development of general software packages is not only of obvious practical significance but can also lead to important theoretical advances. Building nonparametric inference software packages will help us understand where we are faced with fundamental limits and where merely with engineering challenges, just as efforts towards general purpose Bayesian network algorithms led to the discovery of the importance of tree width. While most current algorithms are very model-specific, striving for general purpose methods will help bring about the theoretical framework to discuss these non-parametric models as a family, and the language to describe their various combinations. Last but not least, such software would allow a much larger community to reap the benefits of this research. In return, this field experience would quickly highlight the strengths and weaknesses of current methods, and draw attention to the most pressing needs.

This brings us to the second goal of this workshop. This field attracts researchers from a broad range of disciplines, ranging from theoretical statisticians and probabilists to people building very specialized applications. It is important that we effectively communicate our advances and needs to better focus our effort. Theoreticians need to know which methods are used in practice and why, and what common tricks seem to help, while applied researchers will want to hear about the latest models and inference algorithms. It is especially important to bring statisticians and machine learning researchers together, since these communities work on the closely related topics, but have complementary strengths, often use different terminology, and focus on different application areas.

A key focus of software development, and a top concern of potential users, is scalability. Markov chain Monte Carlo methods have proved their versatility and various advances have greatly improved their speed, but ensuring and assessing convergence remains a difficult topic and it is still unclear whether they will be reliable enough for non experts to use them with confidence. Variational methods, where they have been applied, have brought great speedups and reliable convergence, often at little cost in accuracy, but designing these methods largely remains an art. Additionally, without a better understanding of the loss in accuracy incurred by this approximation, it is possible that this cost will increase in more complex models. This meeting will help us summarize what works, what doesn't, and why, and discuss how to assess performance and build benchmark datasets.

A new workshop is in demand to bring together the growing community around nonparametric Bayesian models and explore these questions. Next we describe how the organization of the workshop will help us achieve the level of interaction we need to make progress, and discuss its relevance to the ICML and UAI communities.

Impact and expected outcomes

Connect communities

Research on nonparametric Bayesian models span a number of communities, and one of the aims of the workshop is to bring together researchers from these communities together, to build ties, exchange ideas, and establish collaborative efforts.

Firstly, we intend to bring together statisticians and machine learning researchers interested in nonparametric Bayesian models. To do so, we have secured funding from the Gatsby Charitable Foundation (and will also apply for PASCAL-II funding) to invite a number of well-respected statisticians to give talks at the workshop, and to participate in the panel and round-table discussions. We will also widely advertise the workshop to other statisticians to encourage attendance. A number of us have attended similar workshops organized within the statistics community and many statisticians have expressed strong interest to get to know the machine learning community better.

Secondly, we intend to bring together both theoreticians interested in developing new models and new inference algorithms, with more applied researchers who might be interested in applying these new models. We hope that by doing so the applied researchers can learn about the newest models and inference techniques, and the theoreticians can be motivated by real world problems to build novel models and design better inference algorithms. We have scheduled a round-table discussion on issues surrounding applications of nonparametric Bayesian models to achieve this exchange. We will also be highlighting some of the success stories of nonparametric Bayesian models in the related applied communities through a number of talks.

Build general purpose software

Over the past year there have been a number of disparate efforts to build general purpose software for nonparametric Bayesian models. We wish to bring together the relevant parties to a round-table discussion to exchange ideas of software design, inference algorithms, and possibly to coordinate efforts to reduce redundant work. A potential outcome here might be a common protocol and language to specify models and inference algorithms, or a common object interface so that models and inference algorithms developed by different parties might be used to build more complex models.

Highlight recent advances

Recent advances in the field include new algorithms (e.g. collapsed variational methods, particle filters), new models (e.g. the Indian Buffet process, and other models based on Beta processes), and new applications (e.g. to signal processing and language modelling). The workshop will serve to highlight all this work to the wider community.

Relevant conferences

The workshop will be of interest to the ICML, UAI and COLT communities.

Nonparametric Bayesian models have found many compelling applications across a number of disciplines addressed by the ICML conference. They have also provided a talking point for a number of current issues in ICML, including model selection, nonparametric modelling and Bayesian vs non-Bayesian approaches.

Many nonparametric Bayesian models are generalizations of the graphical models studied by the UAI community, and are relevant to researchers who are interested in not restricting themselves to parametric graphical models. They have also provided the impetus to develop sophisticated inference algorithms that are applicable to graphical models as well.

There are many unresolved theoretical questions with regard to nonparametric Bayesian models that the COLT community should find interesting. Issues of consistency and rates of convergence fall within the research agendas of many COLT researchers, and these are difficult questions to tackle in the nonparametric Bayesian models context, due to their extremely expressive nature. Many nonparametric Bayesian models also have beautiful representations in terms of online adaptive structures (e.g. the Chinese restaurant process) and these may be of interest to the online learning community.

Schedule

Potential invited speakers

Invitees

References

Organizers

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License