reportSeptember 30 2020

Repairing Innovation

A Study of Integrating AI in Clinical Care

Madeleine Clare Elish
Elizabeth Anne Watkins

Repairing Innovation: A Study of Integrating AI in Clinical Care, by Madeleine Clare Elish and Elizabeth Anne Watkins, reveals the hybrid sociotechnical nature of repair work required to integrate an AI system into clinical health care.

RSVP: Join us today at 5 p.m. ET for an event with Elish, nurse practitioner Dina Sarro, and Innovation Program Manager William Ratliff, who will share their experiences integrating AI systems into hospital operations and celebrate the critical role nurses play to make the system work.

Report Summary

A team at Duke University and Duke Health system developed Sepsis Watch, an AI system that uses deep learning to assess a patients’ likelihood of developing sepsis, to support the Duke Emergency Department with caring for patients with sepsis. Sepsis is a deadly condition that develops from complications with an infection, and while treatable, it can be difficult to diagnose, and early diagnosis is critical.

Elish and Watkins chronicle the integration of Sepsis Watch through a sociotechnical lens: one that acknowledges the human labor required to harmonize a technical system with existing organizational and social structures. The integration of an AI system creates breakages in social structures that must be repaired in order for the technology to work as intended. Rapid Response Nurses at the Duke University hospital took on the bulk of this repair work, according to the authors; work that is often hidden and undervalued. For instance, they mediated professional hierarchies and performed emotional labor to strategically communicate patients’ risk scores to doctors.

Technological systems don’t exist in a bubble. They require a complex interaction of humans, infrastructure, and organizational structure to work effectively. Using a sociotechnical framework, and valuing the work of repair and those who do it, expands our notion of where innovation happens by valuing different forms of expertise and highlighting the importance of different types of workers.

Innovation happens not only when an AI system is being developed, but also when the system is being integrated. This is important to recognize because “when only the work of initiation and theoretical construction–typically elite and masculine forms of work–are valued in AI, then much of the actual day-to-day work required to make AI function in the world is rendered invisible and undervalued.”

Report Authors

Citation

Madeleine Clare Elish and Elizabeth Anne Watkins, Repairing Innovation: A Study of Integrating AI in Clinical Care (New York: Data & Society Research Institute, 2020), https://datasociety.net/pubs/repairing-innovation.pdf.

Connected Track