Neural
Computation: Statistical Perspective
למידה
סטטיסטית
בחישוב עיצבי
0368-4149-01 Autumn
2011-12
Tuesday 16-19 Shenkar
222
Prof. Nathan Intrator
Brief
Abstract
The first part of the course will discuss statistical learning
theory and model estimation as well as issues related to the "Curse of
Dimensionality" and over-fitting. The second part will discuss different
methods for supervised and unsupervised (neural) learning. The third part will
be devoted to model validation methods: model complexity, the variance/bias
dilemma and model ensembles. Several of the latest modeling methods such
as Deep Networks will be presented. Classical methods such as BackProp or Support Vector Machines will also be presented.
Different real-world applications will be discussed throughout the course.
Prerequisites: First Year Calculus, Algebra
and Probability courses.
The Course is open for third year undergraduate students
(sometimes not at the first bidding).
The course has little overlap with courses given by Prof. Brielovski, Yeshurun and Wolf and none of these courses is
a pre-requisite for the current course.
Instructor: Prof. Nathan Intrator, Schreiber 221,
x7598, Office hours: Wednesday 4-5. Contact via email
Partial list of web pointers used
in creation of the lectures
Geoff Hinton
Wlodzislaw Duch
Brian Blais Daniel Silver
Michael Jordan
IAPR
Repository
Other
Useful Pointers
David Horn, Eshel Ben Jacob, Hezy
Yeshurun, Eytan Ruppin, Talma
Hendler (Brain Imaging), NCSP Lab
Brief
Outline
·
Unsupervised
Learning
o
Short bio
motivation
o
Unsupervised
Neuronal Model
o
Connection
with Projection Pursuit and advanced feature extraction
·
Supervised
Learning Schemes
o
Perceptron
and Multi Layer Perceptron
o
RBF, SVM,
Trees
o
Training and
optimization
·
Model
Selection and Validation (advanced training methods)
o
Cross
Validation
o
Regularization
o
Noise
injection
o
Ensembles
·
Brain Machine
Interface
o
EEG, fMRI
modalities
o
Brain state
interpretation based on machine learning model
o
Recent
Research in BMI
Tentative Lectures
- Introduction to neural computation
(Videos
Darpa Grand Challenge2005)
- Additional
Reading: Brain
Facts from the Society for
Neuroscience
- Unsupervised learning:
PCA, Projection Pursuit
- Additional
reading: Theory of Cortical Plasticity Chapter 3, 5;
- Principal
Components Analysis (wiki)
- Hebb
Rule Hebb
Algorithm
- ICA, Measures
of Information, BCM Theory (Draft) fMRI and Mind Reading
- Supplementary:
Introduction
to SVD and EVD
- Independent Components Analysis (wiki)
- Additional
reading: Cover: Information Theory (or other similar books)
- Theory
of Cortical Plasticity Chapter 3.
- BCM
Theory: (Scholarpedia) Exercise
1 Data
for Ex1 (Submission Mon
Dec 26, 2011)
- Physiology
- BCM Learning Rule, Comp
Issues
- PPR and structure
removal
- Additional
material: Plasticity
code, Documentation.
Information
Theory for ML
- Info Theo.
Background Basic
Definitions Objective
Function Formulation of the BCM Theory
- Neuronal Goals:
Efficient Coding and Coincidence Detection
- General
Comments about Paper Writing
- Single Layer
Perceptron
- Multi layer Perceptron
and training methods
- Radial Basis Functions *RBF/Projection Networks
(EM Paper) EM
Wikipedia
- Exercise 2 (Non Linear Regression) Data for Ex2
- Support Vector Machines
- Cross Validation Model
Selection
- Geman et al. Neural networks and the bias/variance dilemma Neural
Com., 4, 1991, 1-58.
- Ensemble Methods (Variance
Bias Decomposition of Ensembles)
- Noise Injection Model Selection (MDL) – Mathematical
Details
- DBN
Deep
Belief Nets Workshop Hinton’s paper Scholarpedia
- Decision
Trees, Introduction
to Bayesian Networks
- (Hidden Markov Models for Speech)
Exam is on Sep 14
Some reading material
|
Neuroscience and Cognitive Science Theory of Cortical Plasticity Cooper et al. Theoretical Neuroscience Dayan & Abbott How are 3D objects represented Bulthoff et al Unsupervised Learning H. Barlow |
Machine learning and Statistics Quick Overview of Books on the topic Pattern recognition and neural networks B. Ripley Neural networks for pattern recognition C. Bishop Online Book based on lectures of Tali Tishby Information Theory T. Cover Introduction to Radial Basis Functions M. Orr Pattern Classification (2nd Edition) Duda, Hart, Stork The Elements of Statistical Learning Hastie, Tibshirani, Friedman Statistics and Neural Networks Kay, Titterington The Bias Variance Dilemma Geman, Bienenstock, Doursat |