B5.2
Tracks
Stream B
| Friday, October 30, 2026 |
| 2:00 PM - 2:15 PM |
Overview
Evaluating AI Scribes in Clinical Workplaces: A Framework for Quality, Readability, and Linguistic Analysis | 15 mins
Presenter
A/Prof. Alireza Ahmadvand
Griffith University
Evaluating AI Scribes in Clinical Workplaces: A Framework for Quality, Readability, and Linguistic Analysis
2:00 PM - 2:15 PMAbstract
The rapid integration of AI-powered clinical documentation tools (AI scribes) into healthcare settings necessitates robust methods for evaluation prior to organisational adoption. Despite increasing use, there remains limited consensus on how to assess the quality, safety, and usability of AI-generated clinical notes.
This presentation introduces a methodological framework developed within the ACE-AI project to evaluate AI-generated documentation across multiple domains. These include note quality using validated instruments (e.g. PDQI-9), readability metrics (e.g. Flesch-Kincaid indices), and linguistic analysis (e.g. lexical diversity and semantic coherence). The framework also considers implications for clinician cognitive load, communication, and decision-making.
Using applied examples, we demonstrate how these measures can be used to compare outputs across AI scribe systems and inform evidence-based procurement and governance decisions in healthcare organisations.
This work contributes to the development of structured, reproducible approaches for evaluating AI tools in professional environments, supporting safer and more effective integration into clinical workflows.
Presentation structure (30 min)
1. Background & Problem (5 min)
• Rapid uptake of AI scribes
• Lack of standardised evaluation frameworks
• Risks: quality, safety, over-reliance
2. Methodological Framework (10 min)
• Quality → PDQI-9 domains
• Readability → Flesch-Kincaid, SMOG
• Linguistic analysis → LSA / lexical diversity
• Reliability → inter-rater considerations
3. Applied Examples (10 min)
• Comparing outputs across tools
• Variability in note quality
• Implications for clinicians and organisations
4. Implications (3–5 min)
• Procurement decisions
• Governance and policy
• Workforce and cognitive load
5. Q&A (5 min)
This presentation introduces a methodological framework developed within the ACE-AI project to evaluate AI-generated documentation across multiple domains. These include note quality using validated instruments (e.g. PDQI-9), readability metrics (e.g. Flesch-Kincaid indices), and linguistic analysis (e.g. lexical diversity and semantic coherence). The framework also considers implications for clinician cognitive load, communication, and decision-making.
Using applied examples, we demonstrate how these measures can be used to compare outputs across AI scribe systems and inform evidence-based procurement and governance decisions in healthcare organisations.
This work contributes to the development of structured, reproducible approaches for evaluating AI tools in professional environments, supporting safer and more effective integration into clinical workflows.
Presentation structure (30 min)
1. Background & Problem (5 min)
• Rapid uptake of AI scribes
• Lack of standardised evaluation frameworks
• Risks: quality, safety, over-reliance
2. Methodological Framework (10 min)
• Quality → PDQI-9 domains
• Readability → Flesch-Kincaid, SMOG
• Linguistic analysis → LSA / lexical diversity
• Reliability → inter-rater considerations
3. Applied Examples (10 min)
• Comparing outputs across tools
• Variability in note quality
• Implications for clinicians and organisations
4. Implications (3–5 min)
• Procurement decisions
• Governance and policy
• Workforce and cognitive load
5. Q&A (5 min)
.....
Dr Alireza Ahmadvand is a General Practitioner and digital health academic based in Brisbane, Australia. He is an Associate Professor in Primary Care at Griffith University and a Senior Medical Officer at Princess Alexandra Hospital. His work focuses on integrating artificial intelligence and digital technologies into primary care to enhance clinical decision-making, medical education, and patient outcomes. Dr Ahmadvand has led research on digital health assisted diagnostics, AI tools for disorders of gut-brain interaction, and the practical use of generative AI in general practice. He actively collaborates across disciplines to drive innovation in healthcare delivery and digital literacy among clinicians. With over 60 peer-reviewed publications, his academic and clinical work aims to translate emerging technologies into real-world benefits for patients and practitioners. Dr Ahmadvand is committed to preparing the next generation of health professionals for a digitally enabled future.
