How should we compare AI mistakes vs human mistakes in healthcare? Doctors make mistakes all the time, but it’s not like we remove the medical license of a doc if they make a few mistakes. The bar for AI is way higher - it has to be nearly perfect and tested in every possible scenario. As soon as it makes a few mistakes at the edges, we feel uncomfortable about putting it into production. It also raises some questions around whether “concordance with what the doctor said” is really the metric to measure. What if it’s being compared to a bad doctor? We probably wouldn’t want them to get the same answers then. There’s probably a few reasons around this. We have a liability framework for doctors making mistakes in the form of malpractice. Doctors have a code of ethics rule, review boards, and social feedback that push them to do the right thing. We’re still figuring out how to deal with liability if AI gets something wrong and what motivates them to do the right thing. Why does mine keep agreeing with me? (Other than the fact that I’m right). But if AI is already better than the bottom quartile of docs who are consistently getting simple cases wrong, we should already be getting this out there? Wait but how do we identify who’s in the bottom quartile of docs?