Readings & resources: A brief history of NLP

L2 (August 27, 2026)
back to schedule

Main reading

• Gururaja, S., Bertsch, A., Na, C., Widder, D., Strubell, E. “To Build Our Future, We Must Know Our Past: Contextualizing Paradigm Shifts in Natural Language Processing.” EMNLP 2023. PDF

A philosophical roots perspective

• Turing, A. M. “Computing Machinery and Intelligence.” Mind LIX(236):433-460, 1950.
• Shannon information theory work, 1950s.
• n-grams; perplexity evaluations.
• Zipf’s Law, 1932.
• Leibniz and Descartes on “universal language”, 1600s: De Arte Combinatoria; Alphabet of Human Thought.

A key people perspective

• Walker, D. E. “Artificial Intelligence and Language Processing: A Directory of Research Personnel.” American Journal of Computational Linguistics, February 1976.

A hardware perspective

• Mark I Perceptron, 1958.
• Minsky, M. and Papert, S. Perceptrons. MIT Press, 1969.
• Raina, R., Madhavan, A., Ng, A. Y. “Large-scale Deep Unsupervised Learning using Graphics Processors.” ICML 2009.

A funding and incentives perspective

• Georgetown-IBM demo, 1954.
• ALPAC. Language and machines: computers in translation and linguistics, 1966. Chair: John Pierce.
• Pierce, J. R. “Whither speech recognition?” 1969.
• Minsky and Papert; the neural AI winter.
• Wayne, C. “Glamour and Deceit”, 1986.

A seminal (empirical) works perspective

• Marcus, M., Santorini, B., Marcinkiewicz, M. A. “Building a Large Annotated Corpus of English: The Penn Treebank.” Computational Linguistics 19(2):313-330, 1993.
• Krizhevsky, A., Sutskever, I., Hinton, G. “ImageNet Classification with Deep Convolutional Neural Networks.” NIPS 2012.
• Mikolov, T., Chen, K., Corrado, G., Dean, J. “Efficient Estimation of Word Representations in Vector Space.” 2013.
• Pennington, J., Socher, R., Manning, C. “GloVe: Global Vectors for Word Representation.” EMNLP 2014.
• Vaswani, A. et al. “Attention Is All You Need.” NIPS 2017.
• Devlin, J., Chang, M.-W., Lee, K., Toutanova, K. “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.” NAACL 2019.
• Radford, A. et al. GPT-2, 2019.
• Brown, T. et al. “Language Models are Few-Shot Learners.” 2020.
• ChatGPT. OpenAI, 2022.

An NLP as a special case of ML perspective

• ALPAC, 1966; Pierce.
• Wayne, C.; the DARPA program and held-out sets.
• Vaswani, A. et al., 2017.

A Bitter Lesson perspective

• Sutton, R. “The Bitter Lesson”, 2019.
• Hoffmann, J. et al. “Training Compute-Optimal Large Language Models.” 2022.
• Smith et al., 2021.

Software setup

• Marimo
• Shell commands
• Python string manipulation