Skip to content

A Primer on Neural Network Architectures for Natural Language Processing

Authors: Collobert, R., Weston, J., Bottou, L., Karlen, M., et al. Year: 2011 ArXiv: https://arxiv.org/abs/1103.0398

Summary

This paper surveys neural network architectures for NLP, including RNNs, CNNs, and their applications. It demonstrates that neural networks can compete with traditional NLP methods and sets the stage for deep learning in NLP.

Key Concepts

  • RNN architectures for sequence processing
  • CNN for NLP feature extraction
  • Word embeddings and distributed representations
  • Multi-task learning in NLP
  • End-to-end differentiable systems

Impact

This work helped establish neural networks as a viable approach for NLP before deep learning's dominance. It influenced the development of end-to-end NLP systems and neural sequence models.

Citation: 

@inproceedings{collobert2011natural,
  title={Natural language processing (almost) from scratch},
  author={Collobert, Ronan and Weston, Jason and Bottou, L{\'e}on and Karlen, Michael and others},
  journal={The Journal of Machine Learning Research},
  volume={12},
  pages={2493--2537},
  year={2011}
}