CS5824/ECE5424: Advanced Machine Learning, Fall 2019 Project Page

Overview

The final project for the Machine Learning course will be a mini research project done by groups of up to 4 people. The overall goal of such a project is to attempt to reproduce and confirm a result from a published machine learning research paper.

Because this is a class project on a tight schedule, it's important to keep your proposed ideas tractable. You should find a research paper that interests you, find an experiment or analysis from the paper, and attempt to repeat the experiment or analysis.

You will not need to submit code for the project, nor are you required to use any particular programming language, platform, or software. You are free to use whatever helps you reproduce the research paper the best. You are welcome to use code provided by authors of the original paper or by others, as long as you take actions to make sure your work attempts to confirm the findings of the original paper (rather than just printing them out again). For example, you could try an experiment on a different train-test split of the data, or on a different dataset altogether.

You may find that it is not possible to reproduce the results from a paper. If that is the case, documenting what you tried could still be valuable, as machine learning research can often be difficult or impossible to reproduce. Reporting what you tried and how you did not obtain the same results as reported in the original paper would be a valuable scientific contribution.

Deliverables

The first deliverable is a project proposal that is due in conjunction with Homework 4.

• Your project proposal should be one page long in a reasonable format. You should use at least 10-point font in a standard typeface such as Times New Roman, Arial, or Helvetica.
• It should discuss
  1. what result from what paper you plan to reproduce,
  2. your preliminary plan for how you will reproduce it, including what software and data you will use, and
  3. how you will evaluate whether you confirmed the original findings.
• Your proposal should list at least three recent, relevant papers you and your team will read and understand as background, including the paper that contains the result you want to confirm.
• It should discuss a proposed plan of who on your team will be responsible for what tasks. These contributions should just be a prediction for planning purposes, and you are not obligated to adhere to them precisely. We just want to make sure you seriously consider division of work early on.

The final deliverables for the class project will be a project website that details your findings with any supplementary material, data, and charts as needed to explain what you tried and what you concluded about whether you can confirm the existing claim.

Your website should have the following elements.

• It should summarize the result from the original paper;
• It should describe the procedure you took to reproduce the result. If you used open-source code, you should link to the code. (Optional) If you created code you're willing to share, provide a link to allow other researchers to see your code.
• It should present measurements and/or analysis of what you discovered when attempting to reproduce the original result.
• Lastly, your website should contain a list of references to relevant papers.

The project website has no length limit, but it should be concise.

Project Ideas

This section is under construction. We will add below some suggested papers with interesting experiments to reproduce. In the meantime, you are encouraged to explore on your own.

Here are some suggested recent papers and ideas on what you could test to verify the papers' findings.

• Understanding Deep Learning Requires Rethinking Generalization. Zhang et al. https://arxiv.org/abs/1611.03530. See if you also zero training loss with random labels given other neural network architectures and other datasets.
• Don't Decay the Learning Rate, Increase the Batch Size. Smith et al. https://arxiv.org/pdf/1711.00489.pdf. Test their claims on a smaller neural network and a smaller dataset, such as the MNIST or fashion-MNIST data.
• On the Convergence of Adam and Beyond. Reddi https://openreview.net/pdf?id=ryQu7f-RZ. See if their method outperforms other optimization methods on a different dataset and model.
• Towards Evaluating the Robustness of Neural Networks. Carlini and Wagner. https://arxiv.org/abs/1608.04644. Test if you get similar sensitivity to adversarial examples if you use a different model architecture or training dataset.
• Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. Bolukbasi et al. https://arxiv.org/abs/1607.06520?context=cs. Test if their debiasing method works for race, age, or some other attribute than gender.
• Sequence to Sequence Learning with Neural Networks. Sutskever et al. http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks. Test if sequence-to-sequence learning works as described, or if is overly sensitive to tuning parameters.
• Wasserstein Generative Adversarial Networks. Arjovsky et al. http://proceedings.mlr.press/v70/arjovsky17a/arjovsky17a.pdf. See if WGANs can learn to generate realistic image from some other domain.

Remember that it's fine to choose a project that has been done by somebody else. The purpose is to try reproducing and verifying someone else's conclusion, so even if you can't come up with a clever variation on the original experiment, just running it again yourself would also be a valuable reproduction for science.