title: Intra- and interobserver agreement with regard to describing adnexal masses using International Ovarian Tumor Analysis terminology: reproducibility study involving seven observers.
abstract: OBJECTIVE
  To estimate intraobserver repeatability and interobserver agreement in assessing the presence of papillary projections in adnexal masses and in classifying adnexal masses using the International Ovarian Tumor Analysis terminology for ultrasound examiners with different levels of experience. We also aimed to identify ultrasound findings that cause confusion and might be interpreted differently by different observers, and to determine if repeatability and agreement change after consensus has been reached on how to interpret 'problematic' ultrasound images.
  
  
  METHODS
  Digital clips (two to eight clips per adnexal mass) with gray-scale and color/power Doppler information of 83 adnexal masses in 80 patients were evaluated independently four times, twice before and twice after a consensus meeting, by four experienced and three less experienced ultrasound observers. The variables analyzed were tumor type (unilocular, unilocular solid, multilocular, multilocular solid, solid) and presence of papillary projections. Intraobserver repeatability was evaluated for each observer (percentage agreement, Cohen's kappa). Interobserver agreement was estimated for all seven observers (percentage agreement, Fleiss kappa, Cohen's kappa).
  
  
  RESULTS
  There was uncertainty about how to define a solid component and a papillary projection, but consensus was reached at the consensus meeting. Interobserver agreement for tumor type was good both before and after the consensus meeting, with no clear improvement after the meeting, mean percentage agreement being 76.0% (Fleiss kappa, 0.695) before the meeting and 75.4% (Fleiss kappa, 0.682) after the meeting. Interobserver agreement with regard to papillary projections was moderate both before and after the consensus meeting, with no clear improvement after the meeting, mean percentage agreement being 86.6% (Fleiss kappa, 0.536) before the meeting and 82.7% (Fleiss kappa, 0.487) after it. There was substantial variability in pairwise agreement for papillary projections (Cohen's kappa, 0.148-0.787). Intraobserver repeatability with regard to tumor type was very good and similar before and after the consensus meeting (agreement 87-95%, kappa, 0.83-0.94). With regard to papillary projections intraobserver repeatability was good or very good both before and after the consensus meeting (agreement 88-100%, kappa, 0.64-1.0).
  
  
  CONCLUSIONS
  Despite uncertainty about how to define solid components, interobserver agreement was good for tumor type. The interobserver agreement for papillary projection was moderate but very variable between observer pairs. The term 'papillary projection' might need a more precise definition. The consensus meeting did not change inter- or intraobserver agreement.
authors.affiliation: Department of Obstetrics and Gynecology, S. Orsola Malpighi Hospital, University of Bologna, Bologna, Italy; Department of Obstetrics and Gynecology, Skåne University Hospital Malmö, Lund University, Malmö, Sweden.
authors.forename: L
authors.initials: L
authors.lastname: Zannoni
authors.forename: L
authors.initials: L
authors.lastname: Savelli
authors.forename: L
authors.initials: L
authors.lastname: Jokubkiene
authors.forename: A
authors.initials: A
authors.lastname: Di Legge
authors.forename: G
authors.initials: G
authors.lastname: Condous
authors.forename: A C
authors.initials: AC
authors.lastname: Testa
authors.forename: P
authors.initials: P
authors.lastname: Sladkevicius
authors.forename: L
authors.initials: L
authors.lastname: Valentin
mesh_terms: D000291:Adnexal Diseases / Q000145:classification / Q000000981:diagnostic imaging* / Q000473:pathology; D000328:Adult; D000368:Aged; D000369:Aged, 80 and over; D003937:Diagnosis, Differential; D005260:Female; D006801:Humans; D021621:Imaging, Three-Dimensional / Q000379:methods; D008875:Middle Aged; D015588:Observer Variation; D011446:Prospective Studies; D015203:Reproducibility of Results; D009626:Terminology as Topic; D018608:Ultrasonography, Doppler / Q000379:methods; D014741:Video Recording
keywords: IOTA; ovarian neoplasms; reproducibility of results; ultrasonography