Abstract / Summary
While AI-conversational agents are increasingly integrated into behavioral and mental health interventions, standardized instruments to assess their performance remain scarce. To address this gap, we developed an instrument to measure user-subjective experience with AI-driven agents in the healthcare domain. We focused on user perception due to its direct contribution to intervention effects, and on healthcare given that the rapid adoption of AI-agents in this domain continues to outpace the development of validated evaluation instruments. Across four studies, we developed and tested the Chatbot User Evaluation instrument (CUE-Chat). First, we identified relevant assessment domains and items through literature review, existing instruments, input from AI-driven agents, and the research team’s domain expertise. Following this, we requested feedback on draft subscales, definitions, and items from expert and laypeople panels. We then tested the structure of the instrument’s concept version. Finally, we conducted a psychometric evaluation and finalized the instrument.CUE-Chat consists of four factors: Perceived Efficacy (4 items), Usability (4 items), Intelligence (4 items), and Safety and Risk Management (4 items). Reliability and validity of the total score and each subscale was good to excellent (Cronbach’s α = .73-.92; Ω = .74-.93). Predictive validity was strong with all scales predictive of users’ overall rating of an AI-agent’s performance.We conclude that CUE-Chat’s strong psychometric properties are promising, and that it is particularly useful for researchers integrating AI-driven agents into behavioral and mental health interventions in both developmental and evaluative stages.