Back to my homepage

---

Course description [syllabus] [preview]

This is an advanced topics course in the machine learning field of classification, with the goals of introducing

1. Dimensionality Reduction
2. Instance-based Methods
3. Discriminant Analysis
4. Logistic Regression
5. Support Vector Machine
6. Kernel Methods
7. Ensemble Methods
8. Neural Networks

all based on the benchmark dataset of MNIST Handwritten Digits.

The course was motivated by a Kaggle competition - Digit Recognizer - and the Fall 2015 CAMCOS at SJSU, and thus has a data science competition flavor. Students will be asked to thoroughly test and compare the different kinds of classifiers on the MNIST dataset and strive to obtain the best possible results for those methods. The use of the MNIST handwritten digits for teaching classification was partly inspired by Michael Nielsen's free online book - Neural Networks and Deep Learning, which notes explicitly that this dataset hits a ``sweet spot'' - it is challenging, but ``not so difficult as to require an extremely complicated solution, or tremendous computational power''. In addition, the digit recognition problem is very easy to understand, yet practically important. To understand more about the rational behind this course, please refer to a reflective linkedin post by the instructor.

---

Course progress

Date	Slides	Further Reading
1/28	Introduction
2/2	Lec 1: Instance-based classifiers	Prof. Veksler's lecture on kNN classification
2/9	Lec 2: Principal Component Analysis	(1) Instructor's notes (2) A tutorial paper
2/18	Lec 3: Discriminant analysis (for dimensionality reduction)	Prof. Veksler's lecture
2/25	Lec 4: Discriminant analysis (for classification)	Prof. Li's lecture
3/8	Lec 5: Two-dimensional LDA	(1) Original paper (2) A note on 2DLDA
3/10	Lec 6: Logistic regression	(1) Prof. Howbert's presentation (2) Andrew Ng's lecture notes
3/22	Lec 7: Support vector machines	(1) Prof. Howbert's presentation (2) Prof. Veksler's lecture (3) A tutorial paper by C. Burges (4) Lagrange duality
4/21	Lec 8: Classification trees and ensemble learning	(1) Trevor Hastie's talk (2) Adele Cutler's lecture
4/26	Lec 9: Neural networks	(1) Prof. Veksler's lecture (2) Free online book by Michael Nielson
5/10	Poster session (introduction)	See Midterm Projects section below
5/12	Oral presentations	See Final Projects section below

Midterm projects (topic summary)

1. Instanced-based classifiers (by Yu Jung Yeh and Yi Xiao)
2. Discriminant analysis classifiers (by Shiou-Shiou Deng and Guangjie He)
3. Two dimensional LDA (by Xixi Lu and Terry Situ)
4. Logistic regression (by Huong Huynh and Maria Nazari)
5. Support vector machine (by Ryan Shiroma and Andrew Zastovnik)
6. Ensemble methods (by Mansi Modi and Weiqian Hou)
7. Neural networks (by Yijun Zhou)

Final projects [instructions]

Some ideas include kernel knn, kernel discriminant analysis, application to USPS handwritten digits, tuning the penalty parameter C in SVM, and deskewing of MNIST digits. Talk to me if you need more ideas or have specific questions.

---

More learning resources

Handling singularity in LDA/QDA

• Psuedoinverse LDA
• Regularized LDA (see Section 4.3.1, page 112 of recommended textbook)
• PCA + LDA
• Direct LDA
• QR + LDA;
• 2DLDA+ LDA;

---

Programming languages

• MATLAB:
  • Common Matlab commands;
  • Online tutorials (see here for a simple one);
  • Statistics and Machine Learning Toolbox Documentation;
• Python:
  • Ryan's lecture;
  • Google's Python Class;
  • kNN in Python
• R:
  • DataCamp's Introduction to R course;
  • kNN in R;
  • PCA in R;

---

Useful course websites

• Prof. Veksler's CS9840a Learning and Computer Vision at University of Western Ontario
• Andrew Ng's CS 229 Machine Learning at Standford University
• Manik's CSL 864 - Special Topics in AI: Classification at Microsoft

---

Data sets

• USPS Zip Code Data
• UCI Machine Learning Repository
• LibSVM data sets
• Extended Yale Face Database B
• Oxford Flowers Category Datasets

---

Instructor feedback

This is an experimental course in data science, being taught at SJSU for the first time. Your feedback (as early as possible) is encouraged and greatly appreciated, and will be seriously considered by the instructor for improving the course experience for both you and your classmates. Please submit your annonymous feedback through this page.

---

Visitor count (since May 12, 2016):
Fast Counters