Introduction: Artificial intelligence (AI) models offer potential benefits in supporting clinical decision-making, diagnosis, and treatment. The study aimed to compare the performance of ChatGPT-4o (Omni) and Gemini Pro in answering clinical questions and case scenarios related to gynecological oncology and to assess the consistency of their long-term responses. Methods: A two-phase comparative analysis was conducted. 700 clinical questions (350 per model) were developed and categorized into open-ended and case-scenario questions. Three months later, the same set of questions was presented again to evaluate any changes in performance for accuracy, completeness, and guideline adherence. Results: Omni outperformed Gemini Pro across all question types (p = 0.001). Omni achieved a mean score of 5.9 for the basic open-ended questions, higher than Gemini, which had 5.1 (p = 0.001). It also maintained a clear advantage in complex, open-ended questions, scoring a mean of 5.6 than Gemini AI’s 4.2 (p = 0.001). Omni scored a mean of 5.7 for basic case scenarios, while Gemini AI lagged with a mean score of 5 (p = 0.001). Omni showed a modest improvement in complex, open-ended queries, with an increase of 0.2 points (+3.57%) (p = 0.001). Omni provided more accurate and comprehensive responses in guideline adherence than Gemini, particularly in complex cases requiring nuanced judgment and adherence to oncology protocols. Its responses aligned with the latest guidelines, including the American Society of Clinical Oncology and the National Comprehensive Cancer Network. Conclusion: Omni is a more reliable and consistent model for answering questions related to gynecological cancers than Gemini. The stability of Omni’s performance over time highlights its potential as an effective tool for clinical applications requiring high accuracy and consistency.

1.
Ubinha
ACF
,
Pedrão
PG
,
Tadini
AC
,
Schmidt
RL
,
Santos
MHD
,
Andrade
C
, et al
.
The role of pelvic exenteration in cervical cancer: a review of the literature
.
Cancers
.
2024
;
16
(
4
):
817
.
2.
Hermann
CE
,
Patel
JM
,
Boyd
L
,
Growdon
WB
,
Aviki
E
,
Stasenko
M
.
Let’s chat about cervical cancer: assessing the accuracy of ChatGPT responses to cervical cancer questions
.
Gynecol Oncol
.
2023
;
179
:
164
8
.
3.
Huang
Y
,
Gomaa
A
,
Semrau
S
,
Haderlein
M
,
Lettmaier
S
,
Weissmann
T
, et al
.
Benchmarking ChatGPT-4 on a radiation oncology in-training exam and Red Journal Gray Zone cases: potentials and challenges for ai-assisted medical education and decision making in radiation oncology
.
Front Oncol
.
2023
;
13
:
1265024
.
4.
Khan
I
,
Khare
BK
.
Exploring the potential of machine learning in gynecological care: a review
.
Arch Gynecol Obstet
.
2024
;
309
(
6
):
2347
65
.
5.
Krückel
A
,
Brückner
L
,
Psilopatis
I
,
Fasching
PA
,
Beckmann
MW
,
Emons
J
.
Evaluation of ChatGPT's potential in tailoring gynecological cancer therapies
.
Vivo
.
2024
;
38
(
4
):
1649
59
.
6.
Levin
G
,
Brezinov
Y
,
Meyer
R
.
Exploring the use of ChatGPT in OBGYN: a bibliometric analysis of the first ChatGPT-related publications
.
Arch Gynecol Obstet
.
2023
;
308
(
6
):
1785
9
.
7.
Levin
G
,
Horesh
N
,
Brezinov
Y
,
Meyer
R
.
Performance of ChatGPT in medical examinations: a systematic review and a meta-analysis
.
Bjog
.
2024
;
131
(
3
):
378
80
.
8.
Sun
X
,
Wei
J
,
Wang
X
,
Wang
B
,
Fan
Z
,
Wan
S
, et al
.
The vast chasm in ChatGPT assisting in realistic surgery
.
Int J Surg
.
2023
;
109
(
11
):
3724
6
.
9.
Grünebaum
A
,
Chervenak
J
,
Pollet
SL
,
Katz
A
,
Chervenak
FA
.
The exciting potential for ChatGPT in obstetrics and gynecology
.
Am J Obstet Gynecol
.
2023
;
228
(
6
):
696
705
.
10.
Stalp
JL
,
Denecke
A
,
Jentschke
M
,
Hillemanns
P
,
Klapdor
R
.
Quality of ChatGPT-generated therapy recommendations for breast cancer treatment in gynecology
.
Curr Oncol
.
2024
;
31
(
7
):
3845
54
.
11.
Yurtcu
E
,
Ozvural
S
,
Keyif
B
.
Analyzing the performance of ChatGPT in answering inquiries about cervical cancer
.
Int J Gynaecol Obstet
.
2025
;
168
(
2
):
502
7
.
12.
Lukac
S
,
Dayan
D
,
Fink
V
,
Leinert
E
,
Hartkopf
A
,
Veselinovic
K
, et al
.
Evaluating ChatGPT as an adjunct for the multidisciplinary tumor board decision-making in primary breast cancer cases
.
Arch Gynecol Obstet
.
2023
;
308
(
6
):
1831
44
.
13.
Johnson
SB
,
King
AJ
,
Warner
EL
,
Aneja
S
,
Kann
BH
,
Bylund
CL
.
Using ChatGPT to evaluate cancer myths and misconceptions: artificial intelligence and cancer information
.
JNCI Cancer Spectr
.
2023
;
7
(
2
):
pkad015
.
14.
Kerbage
A
,
Kassab
J
,
El Dahdah
J
,
Burke
CA
,
Achkar
JP
,
Rouphael
C
.
Accuracy of ChatGPT in common gastrointestinal diseases: impact for patients and providers
.
Clin Gastroenterol Hepatol
.
2024
;
22
(
6
):
1323
5.e3
.
15.
Yeo
YH
,
Samaan
JS
,
Ng
WH
,
Ting
PS
,
Trivedi
H
,
Vipani
A
, et al
.
Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma
.
Clin Mol Hepatol
.
2023
;
29
(
3
):
721
32
.
16.
Liu
J
,
Wang
C
,
Liu
S
.
Utility of ChatGPT in clinical practice
.
J Med Internet Res
.
2023
;
25
:
e48568
.
17.
Fink
MA
,
Bischoff
A
,
Fink
CA
,
Moll
M
,
Kroschke
J
,
Dulz
L
, et al
.
Potential of ChatGPT and GPT-4 for data mining of free-text CT reports on lung cancer
.
Radiology
.
2023
;
308
(
3
):
e231362
.
18.
Pan
A
,
Musheyev
D
,
Bockelman
D
,
Loeb
S
,
Kabarriti
AE
.
Assessment of artificial intelligence chatbot responses to top searched queries about cancer
.
JAMA Oncol
.
2023
;
9
(
10
):
1437
40
.
You do not currently have access to this content.