Chatbots have long been used in medical situations, with mental health-related chatbots increasingly common. A new study from the University of Alabama reminds us that we need to ensure that these technologies are fit for purpose before rolling them out at scale.
The research explores the use of chatbots for providing support to both dementia patients and their caregivers and finds that while they can be beneficial, the variance in quality is significant. For instance, none of the apps tested by the researchers scored highly across all of the criteria, with many containing linguistic biases and usability issues. The researchers believe their findings illustrate the need for far more robust end-user testing by developers.
“Dementia care is complex and no two cases of dementia are alike. Chatbots have the potential of providing caregivers with instant support that is evidence-based and personalized. While it was promising to find some advancements in this area, it was disappointing to learn that more hasn’t been done,” the researchers explain.
Medical chatbots
The researchers suggest that healthcare chatbots certainly have potential, whether in suggesting resources, interpreting symptoms, or offering emotional support. This can be especially valuable for dementia patients as they can face daily challenges. The effectiveness of the chatbots is only as good as the medical knowledge underpinning the chatbot’s logic engine.
“In regards to the use of a chatbot by dementia caregivers, there are many content areas family caregivers may find helpful. Caregivers are confronted by increasing care coordination challenges, and a decline in function and overall well-being of the person living with dementia,” the researchers say. “Additionally, a new caregiver may look for very different resources and information than a more experienced caregiver.”
The researchers identified 27 apps that focused on dementia, and eventually whittled this down to six that fitted the evaluation criteria for analysis. These were: CogniCare mobile app; CogniCare (Alexa Skills version); My Life Story; Dementia Types; Build Your Brain Power; and Everything Memory. Each app was assessed according to its functionality, effectiveness, humanity, and overall satisfaction.
Inadequate support
The analysis found that most of the apps were focused on symptoms and epidemiology rather than caregiving. What’s more, the usability was not great and they were difficult to work with, especially in voice chat mode where extremely precise vocabulary and pronunciation were required for effective conversations to be had. What’s more, each of the chatbots was only available in English, which further limits their usefulness.
The researchers admit that once over these initial teething problems, the apps performed reasonably well, although only one passed the Turing test in terms of the realistic nature of the conversations they had with users. There were also concerns about the ethics and privacy of the apps, with only the two apps produced by CogniCare passing muster.
Ultimately, the researchers found that most of the apps had significant limitations that would hamper the ability of users to have extended and varied conversations with them. Given the complex nature of dementia, this is a serious failure. Similarly, the researchers doubted the robustness of the evidence-base behind the chatbots and how reliable the information provided to users actually is.
While there is undoubtedly some value in chatbots for conditions such as dementia, there is clearly still some way to go before such apps can provide reliable support to either patients or their carers.
