By Rachael Robertson, Enterprise & Investigative Writer
October 30, 2026
As artificial intelligence continues to permeate every facet of modern life, the healthcare sector has found itself at a critical crossroads. Patients, often overwhelmed by the complexities of fertility treatments and the emotional weight of assisted reproductive technology (ART), are increasingly turning to Large Language Models (LLMs) like ChatGPT for guidance. However, a new study presented at the American Society for Reproductive Medicine (ASRM) annual meeting suggests that while these digital assistants offer impressive clarity, they may be falling short in the metrics that matter most: accuracy, safety, and clinical completeness.
The Digital Consultant: Evaluating AI in ART
The research, spearheaded by Nicole Fischer, MD, MPH, a reproductive endocrinology and infertility fellow at the University of Pittsburgh, provides a sobering reality check for the integration of generative AI into patient care. The study set out to answer a fundamental question: When patients seek answers about in vitro fertilization (IVF) and associated risks, are they receiving information that is not only accessible but medically sound and safe?
Using the free version of GPT-5.6 Luna, the research team generated responses to a series of common patient queries regarding the IVF process. These outputs were then rigorously evaluated by a panel of four physicians alongside gold-standard educational materials provided by the ASRM.
Chronology of the Investigation
The investigation was structured into two distinct phases to ensure a comprehensive analysis.
Phase I: Comparative Quality Assessment
The team developed a series of prompts regarding the IVF process and its inherent risks. To ensure consistency, each prompt was processed three times, resulting in a total of 12 responses from the AI. These responses were placed in a blinded review process alongside the official ASRM educational documents. The physicians utilized a 1-5 scale to grade the materials across five key domains: accuracy, completeness, clarity, safety, and overall quality.
Phase II: Quantitative Data and Readability
The second phase focused on the empirical accuracy of the AI. Researchers specifically queried the chatbot on the probability of IVF success across the maternal age spectrum (20 to 45 years). The AI’s prognostications were then cross-referenced with established data from the Centers for Disease Control and Prevention (CDC) and the Society for Assisted Reproductive Technology (SART). Additionally, the team utilized the Flesch-Kincaid scale to determine the reading grade level of the responses, comparing the AI’s output against the 12th-grade readability level typical of ASRM documents.
Supporting Data: Where AI Falters
The results of the analysis paint a stark picture of the limitations inherent in current generative AI models. While ChatGPT was praised for its user-friendly, conversational tone, its performance lagged significantly behind professional medical literature in nearly every other category.
The Scoring Gap
Physicians rated the AI’s responses consistently lower than ASRM materials:
- Accuracy: ChatGPT scored 3.98 compared to the ASRM’s perfect 5.00.
- Completeness: The AI received a 3.40, while professional materials held firm at 4.88.
- Safety: The most alarming disparity appeared here, with the AI scoring 2.92 against the ASRM’s 4.88.
- Quality: The AI trailed with 3.38 compared to the ASRM’s 5.00.
However, the "clarity" metric was the notable outlier. The AI scored a 4.60, nearly identical to the ASRM’s 4.63. This suggests that LLMs are highly proficient at packaging information in a digestible, approachable manner—a feature that may ironically lead patients to trust inaccurate information more readily because it sounds so coherent.
The Misinformation Risk
Perhaps the most concerning finding was the presence of errors in the AI’s output. While the ASRM materials achieved a 100% adherence to key clinical concepts, the AI averaged 94.1%. More critically, 50% of the AI responses contained at least one unsupported claim, and 4.2% included outright misinformation.
Regarding success rates, the AI correctly identified the downward trend of birth rates as maternal age increases. However, the model demonstrated distinct calibration issues, overestimating success rates for 35-year-olds and significantly underestimating them for patients aged 37 and 40.
The Illusion of Fluency: Official Observations
Dr. Fischer, in her presentation, offered a cautionary insight: "The key point here is that fluent, confident wording can coexist with omissions, outdated information, and errors."
This phenomenon, often referred to as "hallucination," occurs because LLMs are designed to predict the most statistically likely word in a sequence, not to verify the factual veracity of the information being generated. In the context of medical advice, this becomes a dangerous liability. "The takeaway here is that topics where accuracy matters most is maybe where these tools are the weakest," Fischer noted.
The study also highlighted the role of "prompt engineering" in patient comprehension. ChatGPT’s responses to conversational prompts were found to be at an eighth-grade reading level, whereas generic prompts yielded a 10th-grade level. While the lower grade level might seem like a benefit for accessibility, the study suggests that the way a patient phrases a question can inadvertently shift the complexity and reliability of the response they receive.
Implications for Future Patient Care
The rise of AI in the doctor’s office—or rather, in the patient’s home—requires a shift in how clinicians manage expectations and guide information seeking.
Bridging the Knowledge Gap
Clinicians are encouraged to take a proactive stance. Rather than dismissing AI, physicians should:
- Engage in Dialogue: Ask patients directly if they are using AI tools to research their fertility journey.
- Teach Digital Literacy: Instruct patients on how to use AI to find reputable citations rather than relying on the chatbot as an primary source of truth.
- Redirect to Trusted Resources: Consistently point patients back to vetted organizations, such as ReproductiveFacts.org or specific clinical literature.
Acknowledging Study Limitations
While the study provides a robust critique of the current state of GPT-5.6 Luna, Dr. Fischer acknowledged inherent limitations. The sample size was relatively small, and the study focused on a single LLM version. With the rapid evolution of specialized tools like "ChatGPT Health," future studies will need to be longitudinal and more expansive to keep pace with the technology. Furthermore, the study measured the quality of the information, not the downstream clinical outcomes or the emotional impact on the patients who consumed the AI-generated advice.
The Future of AI in Medicine
The findings suggest that we are currently in an "experimental phase" regarding the use of AI in reproductive medicine. While these tools excel at synthesis and simplification, they lack the nuanced understanding, ethical grounding, and rigorous fact-checking protocols that define medical expertise.
For now, the consensus remains clear: AI is a powerful tool for generating readable content, but it is not a substitute for the guidance of a board-certified reproductive endocrinologist. As the technology continues to evolve, the medical community must remain vigilant, ensuring that the convenience of an instant, conversational answer does not come at the expense of patient safety or the accuracy of critical health decisions.
In the high-stakes world of fertility, where information is often the only comfort a patient has, ensuring that information is correct is not just a standard—it is a fundamental necessity.
