Literature

NOTE: this list is under construction and subject to change.

Introduction

• Vectorspaces
• Derivatives
• Neural networks
• Machine learning and gradient descent

Recurrent neural networks

• Hochreiter & Schmidhuber (1997): Long Short-Term Memory
• Liu et al. (2018): LSTMs Exploit Linguistic Attributes of Data
• Levy et al. (2018): Long Short-Term Memory as a Dynamically Computed Element-wise Weighted Sum

Additional reading:

• Gers et al. (1999): Learning to Forget: Continual Prediction with LSTM
• Sutskever et al. (2014): Sequence to sequence learning with neural networks.

Convolutional neural networks

• Cireșan et al. (2012): Multi Column Deep Neural Network for Image Classification

Tree-structured recurrent neural networks

• Tai et al. (2015): Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks
• Shi et al. (2018): On Tree-Based Neural Sentence Modeling
• Dyer et al. (2016): Recurrent neural network grammars

Additional reading:

• Nangia & Bowman (2018): Listops: A Diagnostic Dataset for Latent Tree Learning
• Kuncoro et al. (2018): LSTMs Can Learn Syntax-Sensitive Dependencies Well, But Modeling Structure Makes Them Better

Attention

• Bahdanau et al. (2014): Neural Machine Translation by Jointly Learning to Align and Translate
• Vaswani et al. (2018): Attention Is All You Need

Additional reading:

• Luong et al. (2015) Effective Approaches to Attention-based Neural Machine Translation
• Tran et al. (2018) The Importance of Being Recurrent for Modeling Hierarchical Structure

Implementation

• Baydin et al. (2018): Automatic Differentiation in Machine Learning: a Survey

Additional (historical) reading:

• Griewank (1988): On Automatic Di􏰨fferentiation