Controlling for chance agreement in the validation of medical expert systems with no gold standard: PNEUMON-IA and RENOIR revisited

M. Martín-Baranera, J. J. Sancho, F. Sanz

Research output: Contribution to journalArticleResearchpeer-review

6 Citations (Scopus)

Abstract

In the validation of medical expert systems, agreement among different human specialists on a random sample of cases may be taken as a substitute to a missing gold standard. Distance measures between pairs of experts, extensively described in previous studies, do not take into account the influence of chance-expected agreement. A weighted kappa index, with three different weighting schemes, is proposed as an alternative to be applied in the general situation of N cases assessed by E experts about K possible diagnoses, each of them qualified with one of G ordinal categories. A hierarchical cluster analysis, applied to the kappa matrices generated, allows for the classification of the expert system among clinical specialists, providing a relative assessment of its diagnostic ability. The above methodology is applied to the validation of two medical expert systems, PNEUMON-IA and RENOIR. © 2000 Academic Press.
Original languageEnglish
Pages (from-to)380-397
JournalComputers and Biomedical Research
Volume33
Issue number6
DOIs
Publication statusPublished - 1 Jan 2000

Fingerprint

Dive into the research topics of 'Controlling for chance agreement in the validation of medical expert systems with no gold standard: PNEUMON-IA and RENOIR revisited'. Together they form a unique fingerprint.

Cite this