Readings & resources: Evaluation of evaluation

L13 (October 20, 2026)
back to schedule

• Wallach, H. et al. “Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge.” ICML 2025.
• Harvey, E., Koenecke, A., Kizilcec, R. F. “Data Annotation as Measurement.” AIES 2026.
• Fleisig, E., Blodgett, S. L., Klein, D., Talat, Z. “The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels.” 2024.
• Desai, M. et al. “What AI Benchmarks Actually Measure: Adapting Convergent and Discriminant Validity to Interrogate Fifty-Six AI Benchmarks.” 2026.