Sarah Meredith was in urgent need of a liver when she found out an algorithm would be making the life-or-death decision
There are generally two types of livers appropriate for transplantation. One type is donated by people declared “brainstem dead” after catastrophic brain injuries. For decades, these were the only livers considered viable for donation. However, in recent years, livers have increasingly been retrieved from individuals who are not brain-dead, but whose hearts have stopped beating. This is known as a “donation after circulatory death”. Although these now make up about 40 per cent of transplants in the UK, they are known to result in poorer outcomes for recipients, including a higher likelihood of organ rejection and death.
Doctors retain the ability to allocate these latter livers, however the NLOS system is the sole method for allocating the more numerous — and preferred — brainstem death livers. Now, every time one of these livers becomes available anywhere in the UK, the algorithm produces a score for each patient on the waiting list. The score uses 28 variables — seven from the donor and 21 from the recipient — to decide who goes to the top of the list. Essentially, the calculation is the difference in a person’s survival without transplantation (their need) from their survival after transplantation (utility). The highest scoring patient gets the liver, if their doctors want it. If they don’t, the score is recalculated, and the liver is passed on to the new top-scorer, and so on.
The algorithm had been in place for three years when Sarah was put on the waiting list. In an analysis of the algorithm’s outcomes up to that point by the Liver Advisory Group, an advisory panel to the NHS on liver transplantation, the overall number of deaths of individuals on the waiting list had dropped, compared with before the algorithm was introduced. This was treated by the NHSBT as a success.
However, when waiting times were broken down by age, the analysis found that the improvements were primarily for older patients. Patients in Sarah’s age group, 26 to 39 year olds, were waiting far longer than they had previously and considerably longer than older people on the list. Before the algorithm, they remained on the waiting list about 40 days longer than patients over 60. After the algorithm, the gap widened to 156 days.
As Jess began to suspect that Sarah was at a systematic disadvantage under the NLOS system, she sent in a detailed formal complaint and brokered a meeting between her family and the doctors and officials at NHSBT. She had mapped out Sarah’s medical data and brought evidence to show her TBS was not rising above a certain threshold, meaning she was never going to reach the top of the waiting list before she became too ill to transplant. The family wanted to understand what could be done to give Sarah a better chance of getting a liver donation. The meeting, which took place over video, was hugely frustrating for the Merediths. “Every time we brought up the numbers, they would tell us we didn’t understand, presumably because we weren’t doctors,” Jess said. “It’s hard to push back on that.” The medical team also told the Meredith family that no allocation tool was perfect. “They said, ‘we are trying our best but nothing is 100 per cent.’ That wasn’t the point. We just wanted Sarah to have a fair chance,” Jess said.
This was not a problem that the Merediths alone were highlighting. It had been noted by hepatologists across the country, who felt their younger patients were being unfairly disadvantaged. “If you’re below 45 years, no matter how ill, it is impossible for you to score high enough to be given priority scores on the list,” said Palak Trivedi, a consultant hepatologist at the University of Birmingham, which has one of the country’s largest liver transplant centres. Trivedi said patients found this particularly unfair, because younger people tended to be born with liver disease or develop it as children, while older patients more often contracted chronic liver disease because of lifestyle choices such as drinking alcohol. “This is potentially discrimination of a scoring system against young people . . . who have lost a lot of healthy life years,” he said.
Trivedi’s criticisms were threefold. First, he believed the software gave too much weight to older age groups, docking your score if you were under 45. The reasoning behind this was the medical assumption that young people could survive longer than older people, although the long-term effects of waiting longer while chronically ill were unknown. “The disadvantage in . . . [getting] a timely liver transplant if you’re young is too great. So that needs to be revised,” Trivedi said.
Second, he believed the premise of the algorithm — trying to reduce absolute mortality five years after a transplant — was flawed. The system did not account for other outcomes, such as the healthy life years lost by young patients kept waiting, their longer-term outcomes or reduced overall life expectancy. Taking these into account might paint a very different picture of whether the algorithm was beneficial and fair.
Finally, Trivedi said the algorithm was trying to equal out the death rate across all ages on the waiting list, rather than reflecting the rate of the general population, where healthy older people are more likely to die than younger ones. Trivedi believed that transplant patients’ risk should be compared with an age- and sex-matched control population, rather than just against each other.
“The algorithm is intended to allocate available livers to those most likely to benefit, which is an admirable aim. But estimating ‘expected benefit’ is extremely challenging, as there is no data set of comparable patients who did or did not receive transplants,” said David Spiegelhalter, one of Britain’s leading statisticians and an emeritus professor at the University of Cambridge. He added, “A range of subtle statistical issues appear to have unintentionally biased the algorithm against certain classes of patients.”
The problems with the automated liver allocation score were larger than just the statistical flaws. Like many other automated decision-making processes, the algorithm had major human design flaws. For one, the way it was implemented curtailed the agency of human experts, preventing them from challenging its decisions. There was also a lack of transparency in terms of how it worked, and there was no way to appeal exceptional cases, like Sarah’s.
Trivedi and others such as Nigel Heaton, a well-known transplant surgeon at King’s College Hospital, have publicly criticised and lobbied the NHSBT to revise the algorithm. But Trivedi said “that change hasn’t yet happened”.
Olive McGowan, chief nurse at NHSBT, who has worked in organ transplantation and donation for several years, said the algorithm had achieved what it set out to do, “to increase the number of life years gained from transplanted livers and decrease the numbers of people dying on the waiting list”. She said the system had been built by clinicians, statisticians and other experts, and that it was regularly audited to test for unfair outcomes. For instance, last October it was redesigned to correct a bias against transplant patients with liver cancer.
A research team investigating this fatal error showed that for the first three years of the TBS scheme, patients with cancer were rarely allocated a liver by the model. Deaths of patients with cancer on the wait-list increased. The error showed that “algorithms cannot apply common sense”, according to the researchers. Their findings led to revisions to the algorithm.
