title: Methods for semi-automated indexing for high precision information retrieval.
abstract: OBJECTIVE
  To evaluate a new system, ISAID (Internet-based Semi-automated Indexing of Documents), and to generate textbook indexes that are more detailed and more useful to readers.
  
  
  METHODS
  Pilot evaluation: simple, nonrandomized trial comparing ISAID with manual indexing methods. Methods evaluation: randomized, cross-over trial comparing three versions of ISAID and usability survey.
  
  
  METHODS
  Pilot evaluation: two physicians. Methods evaluation: twelve physicians, each of whom used three different versions of the system for a total of 36 indexing sessions.
  
  
  METHODS
  Total index term tuples generated per document per minute (TPM), with and without adjustment for concordance with other subjects; inter-indexer consistency; ratings of the usability of the ISAID indexing system.
  
  
  RESULTS
  Compared with manual methods, ISAID decreased indexing times greatly. Using three versions of ISAID, inter-indexer consistency ranged from 15% to 65% with a mean of 41%, 31%, and 40% for each of three documents. Subjects using the full version of ISAID were faster (average TPM: 5.6) and had higher rates of concordant index generation. There were substantial learning effects, despite our use of a training/run-in phase. Subjects using the full version of ISAID were much faster by the third indexing session (average TPM: 9.1). There was a statistically significant increase in three-subject concordant indexing rate using the full version of ISAID during the second indexing session (p < 0.05).
  
  
  CONCLUSIONS
  Users of the ISAID indexing system create complex, precise, and accurate indexing for full-text documents much faster than users of manual methods. Furthermore, the natural language processing methods that ISAID uses to suggest indexes contributes substantially to increased indexing speed and accuracy.
authors.affiliation: Stanford Medical Informatics, Stanford University, Stanford, California 94035, USA. berrios@email.arc.nasa.gov
authors.forename: Daniel C
authors.initials: DC
authors.lastname: Berrios
authors.forename: Russell J
authors.initials: RJ
authors.lastname: Cucina
authors.forename: Lawrence M
authors.initials: LM
authors.lastname: Fagan
mesh_terms: D000043:Abstracting and Indexing / Q000379:methods*; D001292:Attitude to Computers; D003258:Consumer Behavior; D001330:Electronic Data Processing; D016247:Information Storage and Retrieval; D011795:Surveys and Questionnaires; D013782:Textbooks as Topic; D014584:User-Computer Interface