VOILA! Season 5 episode 4: AI & Science

Seminar page: https://voila-seminars.tilda.ws/tpost/s5e4-ai-science 

Date and time: July 2 2026, 4:00-5:30 PM CEST

Speaker:  Prof. Elliott Ash

Affiliation: ETH Zurich

________________________________________________________________________

Title: Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results 

Topic AI & Science

In this Talk: Recent work has used LLM agents to reproduce empirical social science results with access to both the data and code. We broaden this scope by asking: Can they reproduce results given only a paper's methods description and original data? We develop an agentic reproduction system that extracts structured methods descriptions from papers, runs reimplementations under strict information isolation -- agents never see the original code, results, or paper -- and enables deterministic, cell-level comparison of reproduced outputs to the original results. An error attribution step traces discrepancies through the system chain to identify root causes. Evaluating four agent scaffolds and four LLMs on 48 papers with human-verified reproducibility, we find that agents can largely recover published results, but performance varies substantially between models, scaffolds, and papers. Root cause analysis reveals that failures stem both from agent errors and from underspecification in the papers themselves.

Speaker's BIO: Elliott Ash is Associate Professor of Law, Economics, and Data Science at ETH Zurich's Center for Law & Economics, Switzerland. Prof. Ash is also a Scientific Lead in the Swiss AI Initiative, CEPR Research Affiliate (Political Economy), Associate Editor at Economic Journal, Co-Editor at Journal of Law and Economics, ETH AI Center Core Faculty, and recipient of a European Research Council Starting Grant. Previously, he held research appointments at New York University (Scholar in Residence), University of Warwick (Assistant Professor) and Princeton University (Postdoc). He received a Ph.D. in Economics and J.D. from Columbia University, a B.A. (Plan II Honors) from University of Texas at Austin, and an LL.M. in international criminal law from University of Amsterdam.

 Elliott's research and teaching focus on empirical analysis of the law and legal system using techniques from econometrics, natural language processing, and machine learning. His research has been published in American Economic Journal: Applied Economics, American Economic Journal: Economic Policy, Journal of Law and Economics, Annual Review of Economics, Economic Journal, Cornell Law Review, Georgetown Law Journal, Journal of Public Economics, Journal of Politics, and Political Analysis. Elliott’s research has earned grant funding from the European Research Council, Swiss National Science Foundation, Swiss Data Science Center, U.S. National Science Foundation, the Turing Institute, and the Washington Center for Equitable Growth.