The Dark Side of ChatGPT’s New Medical Records Feature

The latest addition to OpenAI’s ChatGPT platform has been met with a mix of excitement and trepidation. Health in ChatGPT allows users to upload medical records and connect Apple Health data, receiving personalized advice in return.

However, this innovation raises serious concerns about user safety and the risks associated with sharing sensitive information. With hundreds of millions of people turning to the platform every week for health advice, it’s no wonder why AI can seem like a savior in a country where healthcare is often broken. According to Cornell Tech’s Tanzeem Choudhury, “AI can be a democratizing force in healthcare – providing access to information and analysis that might otherwise be out of reach.”

One of the most worrying aspects of Health in ChatGPT is its potential for inaccuracy. We’ve already seen cases where AI has given misleading or even deadly advice. For example, a Florida man sued OpenAI after his life-threatening pulmonary embolism was exacerbated by ChatGPT’s guidance. This sobering reminder highlights that these systems are only as good as their training data.

Even if Health in ChatGPT is equipped with robust safeguards – which is still uncertain – there’s the issue of user consent to consider. Who decides what information gets shared with this chatbot, and how do users know it won’t be used to target them with ads or train even more sophisticated AI models? Columbia University computer science professor Vishal Misra notes that “The same risks exist when sharing health data with anyone – you are sharing something very personal.”

OpenAI has reassured us that physicians around the world helped develop realistic health scenarios and test Health before its release. Additionally, extra encryption protections have been put in place. However, we’ve seen how easily those can be quietly changed or circumvented.

The onus is on users to determine when to escalate concerns to a medical professional. But what happens when that user is elderly or cognitively impaired – and may not have the capacity to make that decision? As Misra cautions, “Whatever it says, take it to an actual human physician. Do not act.”