• Article highlight
  • Article tables
  • Article images

Article History

Received : 22-06-2024

Accepted : 28-06-2024



Article Metrics




Downlaod Files

   


Article Access statistics

Viewed: 123

PDF Downloaded: 54


Get Permission Balasubramanian, Edison, and Surapaneni: Utility of ChatGPT in neuroanesthesia: Still long way to go


Introduction

Dear Editor,

Neuro anesthesiology plays a crucial role in ensuring the safe and effective management of patients undergoing neurosurgical procedures.1 With the advent of artificial intelligence (AI), particularly language models like ChatGPT 3.5, there is a growing interest in evaluating their potential applications in the medical field.2, 3 This study aims to assess the performance of ChatGPT 3.5 in the context of neuro anesthesia by analyzing its responses to a set of clinical cases sourced from a publicly accessible website.

For this study, ten clinical cases as Multiple Choice Questions (MCQ) were selected from the Neuroanesthesia Quiz section of the Society for Neuroscience in Anesthesiology and Critical Care (SNACC) website.4 The cases were diverse, covering a range of neurosurgical scenarios to test the versatility and accuracy of ChatGPT. The methodology involved presenting each case to ChatGPT 3.5 in MCQ format with five options. ChatGPT’s responses were then compared to the correct answers provided in the answer key.4 All responses were generated twice to confirm the answers generated by ChatGPT.

ChatGPT 3.5 demonstrated a moderate level of performance in responding to the neuroanesthesia clinical cases. Out of the ten cases presented, ChatGPT provided correct answers for only four, while the remaining six responses did not match with the key. This suggests a limitation in the model's ability to consistently generate accurate and contextually relevant answers in the domain of neuroanesthesia. Table 1 shows the topics of neuro anesthesiology and the questions in brief which were used to converse with ChatGPT. ChatGPT demonstrated accuracy in topics related to neuroanesthesiology such as position-related complications, and induction agent considerations, postoperative diuresis, fluid and electrolyte management and topical anesthetics, complications in airway management. There were notable inaccuracies in identifying complications during pediatric anesthesia, autonomic responses, cardiac imaging interpretation, air embolism, postoperative complication management, neurosurgical procedures, and ventriculostomy care.

Table 1

Case scenarios inneuro anesthesiology and ChatGPT responses

S. No

Topic

Scenario on

ChatGPT response

1.

Neuroanesthesiology, Position-related Complications.

Quadriplegia after suboccipital craniotomy in the sitting position.

Correct

2.

Pediatric Anesthesia, Surgical Complications.

Severe hypotension and hypoxemia during augmentation cystoplasty.

Incorrect

3.

Autonomic Responses, Vagal Stimulation.

Bradycardia and facial flushing during nephrectomy under general anesthesia.

Incorrect

4.

Pediatric Anesthesia, Intraoperative Crisis Management.

Systolic BP drop and EtCO2 decrease during craniectomy in a 4-month-old child.

Incorrect

5.

Cardiac Imaging, Myocardial Perfusion.

Persistent myocardial filling defect at three hours on a dipyridamole-thallium scan.

Incorrect

6.

Induction Agents, Anesthesia in High-Risk Patients.

Induction agent disadvantage in a patient with carotid artery stenosis.

Correct

7.

Postoperative Diuresis, Fluid and Electrolyte Management.

Increased urine output, serum osmolarity, and hematocrit after occipital astrocytoma resection.

Correct

8.

Air Embolism, Postoperative Complications.

Sudden CVP increase and premature ventricular contractions after craniotomy in the sitting position.

Incorrect

9.

Neurosurgical Procedures, Ventriculostomy Care.

Management of ventriculostomy system during transport after aneurysm coiling.

Incorrect

10.

Topical Anesthetics, Complications in Airway Management.

Cyanosis and decreased SpO2 after topical benzocaine for fiberoptic intubation

Correct

The findings of this study carry significant implications for the potential integration of language models like ChatGPT 3.5 into the complex landscape of medical decision-making, specifically within specialized fields such as neuroanesthesia. While ChatGPT demonstrated a moderate level of competency by providing accurate responses for a few of the cases, the notable limitation of being unable to deliver accurate answers in more than half of the scenarios raises substantial concerns about its reliability, particularly in critical medical situations where precision is paramount.

The inability of ChatGPT 3.5 to consistently generate accurate responses suggests a need for further research and refinement before considering its practical deployment in neuroanesthetic decision-making processes. Neuro anesthesiology involves intricate scenarios with nuances that demand a high level of contextual understanding, and the model's limitations in this regard underscore the complexity of translating language models into reliable tools for specialized medical domains.5, 6

To enhance ChatGPT’s overall performance, future research efforts should focus on targeted training and refinement tailored to the specific challenges posed by neuro anesthesiology cases. This might involve incorporating more recent and diverse medical data, as well as collaborating with domain experts to fine-tune ChatGPT’s responses to the intricacies of real-world scenarios. Additionally, exploring methods to provide real-time interaction and feedback mechanisms could contribute to refining the model's accuracy and addressing its limitations in critical medical contexts.

Conclusion

This study provides an initial assessment of ChatGPT 3.5 in the field of neuro anesthesiology, revealing both its strengths and limitations. While the model demonstrated some capability in answering clinical cases, its accuracy fell short in a significant portion of scenarios. However, this study is subjected to limitations as only 10 clinical cases were used. More comprehensive assessments are required to generalize the findings. Future iterations and improvements in training methodologies may be required to enhance the model's performance and reliability for practical clinical applications in neuro anesthesiology. This research contributes to the ongoing dialogue surrounding the integration of AI in healthcare and underscores the importance of rigorous evaluation before widespread implementation in specialized medical domains.

Sources of Funding

Nothing to declare.

Conflict of Interest

None to be declared.

Acknowledgements

Authors would like to extend their gratitude to the OpenAI, an American artificial intelligence research laboratory for providing free access to ChatGPT 3.5.

References

1 

Y Longhitano C Zanza The Route of Neuro-Critical CareRev Recent Clin Trials20221742256

2 

J Liu C Wang S Liu Utility of ChatGPT in Clinical PracticeJ Med Internet Res202325e48568

3 

RA Khan M Jawaid AR Khan M Sajjad ChatGPT - Reshaping medical education and clinical managementPak J Med Sci20233926057

4 

Society for Neuroscience in Anesthesiology and Critical Care (SNACC)https://snacc.org/education-corner/neuroanthesia-quiz/Last accessed on 30-11-23

5 

M Sallam ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid ConcernsHealthcare (Basel)2023116887

6 

M Cascella J Montomoli V Bellini E Bignami Evaluating the Feasibility of ChatGPT in Healthcare: An Analysis of Multiple Clinical and Research ScenariosJ Med Syst202347133



jats-html.xsl


This is an Open Access (OA) journal, and articles are distributed under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 License, which allows others to remix, tweak, and build upon the work non-commercially, as long as appropriate credit is given and the new creations are licensed under the identical terms.