Nikhil Agarwal (MIT): Combining Human Expertise with Artificial Intelligence

The webinar is on November 30, 9-10am Pacific Time (12-1pm ET)

Presenter: Nikhil Agarwal (MIT)

Discussant: Ziad Obermeyer (UC Berkeley)

Zoom webinar link: https://stanford.zoom.us/j/92769922331?pwd=NlpQWWZvbFFJdUJMYnZsRWJ6bVZG…

Webinar ID: 927 6992 2331, Passcode: 847333

For more information, please visit our website: https://www.abfr-forum.org

To stay up to date please join our mailing list: https://groups.google.com/u/0/g/abfr-forum

Combining Human Expertise with Artificial Intelligence

Experimental Evidence from Radiology

Authors: Nikhil Agarwal (MIT), Alex Moehring (MIT), Pranav Rajpurkar (Harvard) and Tobias Salz (MIT)

Abstract: While Artificial Intelligence (AI) algorithms have achieved performance levels comparable to human experts on various predictive tasks, human experts can still access valuable contextual information not yet incorporated into AI predictions. Humans assisted by AI predictions could outperform both human-alone or AI-alone. We conduct an experiment with professional radiologists that varies the availability of AI assistance and contextual information to study the effectiveness of human-AI collaboration and to investigate how to optimize it. Our findings reveal that (i) providing AI predictions does not uniformly increase diagnostic quality, and (ii) providing contextual information does increase quality. Radiologists do not fully capitalize on the potential gains from AI assistance because of large deviations from the benchmark Bayesian model with correct belief updating. The observed errors in belief updating can be explained by radiologists’ partially underweighting the AI’s information relative to their own and not accounting for the correlation between their own information and AI predictions. In light of these biases, we design a collaborative system between radiologists and AI. Our results demonstrate that, unless the documented mistakes can be corrected, the optimal solution involves assigning cases either to humans or to AI, but rarely to a human assisted by AI.

Bio of speaker: Nikhil Agarwal is a Professor of Economics at the Massachusetts Institute of Technology. Professor Agarwal studies the design of markets and the interface between economic theory, statistics and real-world practice. He has applied his work to markets in education and healthcare settings, including organ donations, residency matching and school choice. He has published several articles at top journals such as Econometrica and the American Economic Review. His research is funded by the NIH and NSF. He is a recipient of the Sloan Research Fellowship. He received his PhD in Economics from Harvard.

Bio of discussant: Ziad Obermeyer is Associate Professor and Blue Cross of California Distinguished Professor at UC Berkeley. His research uses machine learning to help doctors make better decisions, and help researchers make new discoveries by “seeing" the world the way algorithms do. His work on algorithmic racial bias has impacted how many organizations build and use algorithms, and how lawmakers and regulators hold AI accountable. He is a cofounder of Nightingale Open Science and Dandelion Health, a Chan Zuckerberg Biohub Investigator, a Faculty Research Fellow at the National Bureau of Economic Research, and was named one of the 100 most influential people in AI by TIME magazine. Previously, he was Assistant Professor at Harvard Medical School, and continues to practice emergency medicine in underserved communities.