Deep Learning

Fall Semester 2019

Deep learning is an area within machine learning that deals with algorithms and models that automatically induce multi-level data representations.

In recent years, deep learning and deep networks have significantly improved the state-of-the-art in many application domains such as computer vision, speech recognition, and natural language processing. This class will cover the fundamentals of deep learning and provide a rich set of hands-on tasks and practical projects to familiarize students with this emerging technology.

Course Catalogue Info

News

14.11.2019 Please see updated exercise shedule below.
09.09.2019 The webpage for this year's course is online.

Course Overview

WeekDateTopicSlidesExercisesMaterial
1 19.09.2019 Overview of course, Intro to neural networks, MLP intro & admin slides1 Meta-introduction to Tensorflow exercise 1 solution 1
2 26.09.2019 Feedforward DNNs slides2 tutorial 2 exercise 2
solution 2
3 03.10.2019 Backpropagation slides3 tutorial 3 exercise 3
solution 3
- 10.10.2019 NO CLASS
4 17.10.2019 Convolutional Neural Networks, CNNs in Computer Vision slides4 blackboard exercise 4
solution 4
5 24.10.2019 CNNs for Natural Language, Recurrent Neural Networks slides5 tutorial 5 exercise 5
solution 5
exercise 5 code/model
exercise 5 data
solution 5 code
6 31.10.2019 Memory & Attention slides6 exercise 6
solution 6
optimization.ipynb
optimization-solution.ipynb
7 07.11.2019 Optimization slides7 exercise 7
solution 7
8 14.11.2019 Deep Optimization & Theory slides8
9 21.11.2019 Deep Learning Theory slides9 exercise 8 with solutions
10 28.11.2019 DL on Graphs, Autoencoder, Factor Analysis slides10 exercise 9 with solutions
11 05.12.2019 Generative Models slides11 exercise 10
solution 10
- 12.12.2019 NO CLASS (NeurIPS) BUT exercise session! exercise 11
solution 11
12 19.12.2019 Other, Outlook slides12

Projects

30% of your final grade will be determined by a group project. Form groups of 3-4 students (use piazza if you can't find any teammates). The topic of the project can be freely chosen as long as it has a practical component (code) in the space of deep learning and is of scientific nature (i.e. answering a scientific question rather than pure engineering).

The project grade is composed by 10% of your project proposal grade and by 90% of your final hand-in grade.

Project Proposal

Send a project proposal until December 23rd 2019 to eth_deep_learning@outlook.com using the subject line [Project Proposal] <your project title> and the following template as body:

Names: <comma separated list of author names>
Nethz: <comma separated list of nethz accounts>
Email: <comma separated list of email addresses>
IDs: <comma separated list of student ID numbers (XX-XXX-XXX)>

attach your proposal as a 1-page PDF (excluding references) to the email.
Proposal Grading. The proposal will be graded in equal weight by the following two criteria: 1.) Clarity of idea, clear description of methods, datasets, procedure, baselines, etc. 2.) Presentation of a literature overview over your chosen task.

Project Hand-In

Hand in your report as a PDF with a maximum of 5 pages (additional pages can be used for listing references) and hand in your source code and other supplementary material along with a README file explaining the material and explaining how to run your code to reproduce your experiments to https://cmt3.research.microsoft.com/ETHZDL2019 until January 18th 2020, 5PM Pacific Time.

Report Structure
The structure should be:
1) Introduction
Describe your problem and state your contributions.
2) Models and Methods
Describe your idea and how it was implemented to solve the problem. Survey the related work, giving credit where credit is due.
3) Results
Show evidence to support your claims made in the introduction. Compare to baselines / existing work.
4) Discussion
Discuss the strengths and weaknesses of your approach, based on the results. Point out the implications of your novel idea on the application concerned.
5) Summary
Summarize your contributions in light of the new results.

Report Grading
Your paper will be graded in equal weights according to the following three criteria:

1) Quality of paper
----------------

6.0: Good enough for submission to an international conference.
5.5: Background, method, and experiment are clear. May have minor
issues in one or two sections. Language is good. Scores and baselines are well documented.
5.0: Explanation of work is clear, and the reader is able to identify the novelty of the work. Minor issues in one or two sections. Minor problems with language. Has all the recommended sections in howto-paper.
4.5: Able to identify contribution. Major problems in presentation of results and or ideas and or reproducibility/baselines.
4.0: Hard to identify contribution, but still there. One or two good sections should get students a pass.
3.5: Unable to see novelty. No comparison with any baselines.


2) Execution of idea
----------------------

6.0: Idea is executed well, everything in the proposal has either been followed or changed according to feedback or a reason is provided why it was altered. The project can go beyond the proposal, guided by results obtained as a consequence of executing it. The experiments done make sense in order to answer the proposed research questions. There are no obvious experiments not done that could greatly increase clarity. The submitted code and other supplementary material is understandable, commented, complete, clean and there is a README file that explains it and describes how to reproduce your results.

Subtractions from this grade will be made if:
- the submitted code is unclear, does not run or experiments cannot be reproduced or there is no description of it
- plans made in the proposal have been severely altered or ignored without provided reason
- experiments done are useless to gain understanding or of unclear nature or obviously useful experiments have been left undone
- comparison to baselines are not done


3) Creativity of solution
----------------------

6.0: Elegant proposal, either making a useful assumption, studying a particular class of data, or using a novel mathematical fact.
5.5: A non-obvious combination of ideas presented in the course or published in a paper (Depending on the difficulty of that idea).
5.0: A novel idea or combination not explicitly presented in the course.
4.5: An idea mentioned in a published paper with small extensions / changes, but not so trivial to implement.
<=4.0: A trivial idea taken from a published paper.

Compute Resources

Once you get feedback on your proposal, you will have access to ETH's Leonhard cluster

Early Submission Deadlines

Submit the proposal by December 14 to get 0.25 extra on your proposal grade. Note that this means you could get a 6.25 on the proposal.
Submit the project by January 5 to get 0.25 extra grade on your project grade. Unlike the proposal bonus, this will be subject to capping, so no grades higher than 6 for the project.

Past exams

Exam 2016 Exam 2017

Testat

In order to obtain the Testat, you need to pass the project with grade 5 or higher.
No exam is required.

Contact

To ask a questions, please do not send emails but ask on our forum at piazza. The lecture is 263-3210-00L, you can sign up here. Please post questions there, so others can see them and share in the discussion.

Lecturer Thomas Hofmann
Head TA Jonas Kohler
Head TA Yannic Kilcher
TA Florian Schmidt
TA Hadi Daneshmand
TA Javier Montoya
TA Paulina Grnarova
TA Kevin Roth
TA Leonard Adolphs
TA Matthias Hüser
TA Antonio Orvieto
TA Dario Pavllo
TA Janis Fluri
TA Yeyao Zhang

Literature

Deep Learning, An MIT Press book, Ian Goodfellow, Yoshua Bengio and Aaron Courville