Abstract
Introduction: Artificial intelligence (AI) models offer potential benefits in supporting clinical decision-making, diagnosis, and treatment. The study aimed to compare the performance of ChatGPT-4o (Omni) and Gemini Pro in answering clinical questions and case scenarios related to gynecological oncology and to assess the consistency of their long-term responses. Methods: A two-phase comparative analysis was conducted. 700 clinical questions (350 per model) were developed and categorized into open-ended and case-scenario questions. Three months later, the same set of questions was presented again to evaluate any changes in performance for accuracy, completeness, and guideline adherence. Results: Omni outperformed Gemini Pro across all question types (p = 0.001). Omni achieved a mean score of 5.9 for the basic open-ended questions, higher than Gemini, which had 5.1 (p = 0.001). It also maintained a clear advantage in complex, open-ended questions, scoring a mean of 5.6 than Gemini AI’s 4.2 (p = 0.001). Omni scored a mean of 5.7 for basic case scenarios, while Gemini AI lagged with a mean score of 5 (p = 0.001). Omni showed a modest improvement in complex, open-ended queries, with an increase of 0.2 points (+3.57%) (p = 0.001). Omni provided more accurate and comprehensive responses in guideline adherence than Gemini, particularly in complex cases requiring nuanced judgment and adherence to oncology protocols. Its responses aligned with the latest guidelines, including the American Society of Clinical Oncology and the National Comprehensive Cancer Network. Conclusion: Omni is a more reliable and consistent model for answering questions related to gynecological cancers than Gemini. The stability of Omni’s performance over time highlights its potential as an effective tool for clinical applications requiring high accuracy and consistency.
