Over the last few years, there has been a multitude of possible use cases presented for the use of AI in healthcare, from disease identification to ICU optimization.  Research from Stanford’s Graduate School of Business suggests that a major barrier to achieving these use cases is securing buy-in from users, which in healthcare includes both patients and medical staff.

“Machine learning is a technology that’s not well understood and thus not well trusted,” the researchers say. “People describe it as a black box — they feel like they don’t have input into how it’s used. It has an ability to add value, but only if we can create trust.”

They believe that a more collaborative process between the tech developers and front-line staff can help to overcome these problems, and in the study showed how it could work in two hospitals.

“This work shows improvement to the design of the machine learning tools based on user input in real-world settings, which just hasn’t been done before,” they explain.

Co-creating

Despite significant progress in agile software development, the authors argue that the majority of software still tends to be developed by the tech team who then require users to adapt their practices to the tool.  This obviously has implications for the ease with which such tools can be integrated into workflows, not to mention the utility of the tool to begin with.

The researchers conducted interviews with the developers behind a couple of AI-based apps that were being designed to improve the accuracy of bed availability predictions in ICU and of the risk of readmission for patients respectively.

This interview process revealed clear examples of where the developers had managed to engage end-users and solicit their feedback into the tools and how they were working so that this feedback could be introduced into the next iteration of the tool.

“In any kind of group where you have people with very different professional backgrounds, it can be hard for one side to hear from someone sitting across the table about what they’ve done wrong or what they haven’t considered,” the researchers conclude. “Trust came over time. If a developer listened to what the user was saying, the user gained trust in what developers gave back to them. It was remarkable in the way that it enabled users to then say, ‘I need to consider this.’”

While one would hope that agile software development was the norm in healthcare, just as it is in other sectors, but if it’s not, this research might provide an additional reason why it should be.