The role of a physician involves constant evaluation and reassessment of various factors such as the success rate of medical procedures, the risk of severe symptoms in patients, and the timing for further testing. With the advancement of artificial intelligence (AI), there is a promising opportunity to reduce risks in clinical settings and assist physicians in prioritizing the care of high-risk patients.
However, researchers from the MIT Department of Electrical Engineering and Computer Science (EECS), Equality AI, and Boston University are advocating for more oversight of AI by regulatory bodies. This call for regulation comes after the U.S. Office for Civil Rights (OCR) of the Department of Health and Human Services (HHS) issued a new rule under the Affordable Care Act (ACA) in May. The rule prohibits discrimination based on various factors in “patient care decision support tools,” which include both AI and non-automated tools used in medicine.
The final rule, developed in response to President Joe Biden’s Executive Order on AI development and use, aims to prevent discrimination and promote health equity. Senior author Marzyeh Ghassemi, an associate professor of EECS, emphasizes the importance of this rule in driving equity-driven improvements in clinical decision-support tools.
Despite the increasing number of FDA-approved AI-enabled devices, there is a lack of regulatory oversight for clinical risk scores produced by decision-support tools. The Jameel Clinic at MIT is taking steps to address this issue by hosting a regulatory conference in March 2025 to discuss the regulation of AI in health.
Isaac Kohane, chair of the Department of Biomedical Informatics at Harvard Medical School, highlights the importance of holding clinical risk scores to the same standards as AI algorithms. Even non-AI decision-support tools can perpetuate biases in healthcare and require oversight to ensure transparency and non-discrimination.
Maia Hightower, CEO of Equality AI, acknowledges the challenges in regulating clinical risk scores embedded in electronic medical records. She emphasizes the necessity of regulation to prevent discrimination and ensure transparency in healthcare practices.
As the regulation of clinical risk scores faces challenges under the current administration, researchers stress the importance of maintaining a focus on equity and non-discrimination in health care. The article published in the New England Journal of Medicine AI provides valuable insights on the need for regulatory oversight in the use of AI and decision-support tools in healthcare.
