Commentary: What you need to know about AI’s role in your A&E visit

SINGAPORE: Imagine this scenario: You’re in a crowded emergency room in one of Singapore’s hospitals. Around you, dozens of patients are waiting to be triaged and seen by a doctor. The flow of incoming patients is constant: Some arrive by ambulance, their conditions appearing critical and dire, while others, with more minor complaints, wait to be seen in the waiting room.

In the midst of this, a triage scoring tool built upon an AI model specifically for use in the emergency department works behind the scenes, helping medical staff decide which patients should be seen first.

The doctor in charge of the triage process makes a professional judgment about how quickly the patient should be seen and what treatment should be instigated, and then consults the AI-produced triage score to consider whether it would be appropriate to modify their plan in any way. Here, the AI model is “in the loop” of human decision-making, but it is the doctor who is calling the shots. The AI model is simply supporting the doctor’s professional judgment.

Consider a second, slightly modified scenario where the doctor is mere technician, putting in place a triaging plan entirely based on the score produced by the AI model, free from any professional judgment.

Which of these two approaches is to be preferred will, in part, depend on how effective the AI triage scoring tool is compared to the judgment of expert clinicians. But both approaches also raise a further question: Do patients have the right to know the role the AI model is playing in critical decisions about how long they must wait to see a doctor?

AN OBLIGATION TO TELL?

We expect many readers will think that when patients place decisions about their health (partly) in the hands of a machine, they should be informed about this fact. But is this view correct?

In Singapore, the legal requirement to disclose information to a patient in the consent process focuses on the appropriate exercise of "peer professional" judgment that incorporates two key requirements:

• disclosing information that a person in the same circumstances as the patient would reasonably require to make an informed decision, and
• disclosing information that the healthcare professional knows, or ought reasonably to know, is material to the patient for the purpose of making an informed decision

Does this mean that the doctor in the scenarios above is required, upfront, to inform the patient that the AI model is contributing to or making the decision? We think not.

Ultimately, it is the principle of respecting a patient’s autonomy that should primarily shape the information-giving requirements of any consent process in a healthcare setting.

The value of autonomy gives rise to an ethical obligation to provide information that materially affects any specific patient’s considered judgment about what treatment to accept, in light of that patient’s personal values and preferences. Here, the professional’s obligation is to tell the patient about the things that matter to them, in so far as this comes to be known in dialogue between the two parties.

But, we do not typically require healthcare professionals to provide a detailed account of every step underpinning their clinical recommendations, or to explain the precise decision-making pathway involved.

Consider some comparable examples: Doctors commonly seek the view of colleagues before making a treatment recommendation. At other times, they seek out information from an online resource or from the conclusions presented in a recent research paper to finalise their treatment plan.

In these examples, a doctor could have relied entirely on these other sources to determine their recommendation, in just the same way as they might rely on the output from the AI model. In such scenarios, we do not typically expect doctors to explain to patients how and why they have devised these recommendations.

Why is this? As explained above, the principle of respect for autonomy does not obviously demand it: Such detailed background information is unlikely to be of critical importance to patients in formulating their considered view on the course of action they would choose or accept.

Secondly, two other ethical principles - beneficence and non-maleficence - also come into view. Aside from respecting patients’ autonomy, doctors are required to take appropriate steps to act in ways that provide benefits to their patients and that do not cause harm. Providing overly detailed or extraneous information could itself disadvantage patients, perhaps because it might take too much time to disclose, disorientate them, or undermine their trust in the doctor.

In our view, there is nothing obvious in this ethical discussion that justifies changing existing information disclosure practices simply because an AI model is shaping or determining the doctors’ actions - in the absence of compelling population-level evidence that widespread worries, fears, misunderstandings, or mistrust exists about the varied roles that AI could play in healthcare decision-making.

WHEN WOULD DETAILED INFORMATION BE REQUIRED?

It is worth reiterating that the principle of respect for autonomy should not be interpreted in such a way that it propagates a defensive approach to information disclosure, in which healthcare professionals aim at telling patients everything about their treatment plan (and how it was formulated) just in case one patient under their care would have wanted to know about it.

It is the specific tailoring of information in response to the concerns expressed by each patient that is legally and ethically required, not a broad scattergun approach.

However, we can imagine a slightly different scenario where a doctor is posed a direct question by a patient in the waiting room, who is perhaps growing frustrated that they are waiting longer than other patients to be taken to a treatment room: “Why am I having to wait so long to be seen? How did you decide that I had to wait this long?”

Here, the ethical requirement to respect autonomy would indeed require the doctor to provide more detail about the process by which that judgment was made, and the role that the AI model played in this.

The doctor would also need to be equipped to address any follow-up questions about how the AI model operates and/or why it can be relied upon. This is because the patient has made an explicit request to know how a decision about the management of their care has been determined, and this decision is relevant to any further choice they will then make about whether they wait longer or not.

RECONCILING NEW RESPONSIBILITIES WITH ESTABLISHED DUTIES

Singapore is an ageing population where one in four will be aged 65 and above by 2030. The healthcare sector is facing a manpower crunch, with nurses in short supply and hospitals struggling to meet demand for critical roles such as radiographers and pharmacists.

In outlining considerations for the regulation of AI in healthcare on Oct 19, the World Health Organization said that if understood properly, AI tools “could transform the health sector”. These tools offer improvements to medical diagnosis, treatment and self-care, supplementing healthcare professionals’ knowledge, skills and competencies.

We might quibble over whether AI represents a revolution or evolution in healthcare practice. However, we can all agree that the deployment of AI models by the bedside is beginning to bear fruit. There are now numerous examples of how machine learning algorithms with impressive predictive power comprise an important new weapon in the healthcare professional’s arsenal to improve quality of care. The reliance on AI models in delivering healthcare is only likely to increase.

Important responsibilities accompany these changes; the public should be kept updated in general terms about how the health system is being transformed to benefit them more effectively and efficiently. Other challenges that AI models give rise to when deployed in healthcare must also be addressed, including risks of bias and uncertainties regarding professional, legal and ethical accountability for errors.

However, it does not follow from meeting these general requirements that an overhaul of established information disclosure practices in patient care is needed. When it comes to determining what information that should be shared with patients, AI is not exceptional.

Dr Michael Dunn is Associate Professor, Centre for Biomedical Ethics, Yong Loo Lin School of Medicine, National University of Singapore. Dr Liu Nan is Associate Professor, Centre for Quantitative Medicine, Duke-NUS Medical School.