AAA benchmarking by Dr Foster: a cause for concern?
BACKGROUND
The publication of interpretable performance data for hospitals is an important service. In November 2002, the medical benchmarking company Dr Foster published a league table based on the results of abdominal aortic aneurysm (AAA) repair. The purpose of our study was to establish the validity of the data used in benchmarking.


METHODS
Data on elective infra-renal AAA (IRAAA) repair was obtained from three sources. Data used by Dr Foster was based on the hospital PAS system. The databases for both Dr Foster and PAS were analysed and cross-referenced to the vascular unit database maintained by a separately employed audit co-ordinator.


RESULTS
Of 395 total aortic aneurysm repairs, 223 (56%) were identified as elective IRAAA repairs on the unit database. Of these, 125 were identified on the PAS database and 115 on the Dr Foster database. The number of deaths was the same in both the unit and Dr Foster databases (n = 11) but the Dr Foster database included deaths in patients who had undergone juxtarenal (n = 1), Type III TAAA (n = 2) and Type IV TAAA (n = 4) repairs and omitted 7 deaths following IRAAA. The sensitivity and specificity for the PAS dataset was 0.51 and 0.93, respectively. For Dr Foster, the results were worse with a sensitivity and specificity of 0.41 and 0.86, respectively. The accuracy of the data was 0.6 and 0.69 for Dr Foster and PAS, respectively. Standardised mortality ratios (SMRs) were used to rank hospitals. Dr Foster's published SMR for elective AAA repair for our unit was 160. The actual SMR was 67.


CONCLUSIONS
Robust and accurate published league tables should be supported and commended but currently available data appear to be misleading and may cause unnecessary concern to patients.
Pandey
V A
VA
Regional Vascular Unit, St Mary's Hospital, London, UK.
Kerle
M I
MI
Jenkins
M P
MP
Wolfe
J H N
JH
D000368:Aged; D017544:Aortic Aneurysm, Abdominal / Q000401:mortality / Q000601:surgery*; D019985:Benchmarking / Q000592:standards*; D019992:Databases as Topic / Q000592:standards; D004630:Emergencies / Q000453:epidemiology; D004638:Emergency Treatment / Q000592:standards / Q000706:statistics & numerical data; D005568:Forms and Records Control / Q000592:standards; D006801:Humans; D007902:Length of Stay / Q000706:statistics & numerical data; D008131:London / Q000453:epidemiology; D008875:Middle Aged; D012680:Sensitivity and Specificity; D014656:Vascular Surgical Procedures / Q000592:standards* / Q000706:statistics & numerical data