Impact of inter-judge agreement on perceptual judgment of nasality.
OBJECTIVE
To investigate the effect of perceptual inter-judge agreement of hypernasality on velopharyngeal (VP) closure prediction.


METHODS
Two logistic regression models were developed aiming to verify the possibility of predicting the VP closure using the following characteristics: rating of VP closure (adequate, borderline, inadequate), determined by the pressure-flow technique, degree of hypernasality (absent, mild, moderate, severe), and the presence/absence of nasal air emission and nasal rustle determined perceptually by three experienced speech language pathologists. In the first model, 100 speech samples with a moderate agreement rate of hypernasality (kappa coefficient: 0.41) were used. In the second model, 43 speech samples with a perfect agreement among judges were included. The χ²-test was used to compare the models (p≤0.05).


RESULTS
In the first model, 65 of the 100 samples were rated in the correct VP closure category, with 42 adequate and 23 inadequate. The borderline VP closure was not predicted. The second model rated 31 of the 43 samples in the correct category, with 21 adequate VP closure, 5 in the borderline VP closure, and 5 inadequate. There was no difference (p=0.526) between the two models. However, the second model showed a higher proportion of accuracy (7%) than the first one, and it has also predicted the borderline VP closure.


CONCLUSIONS
These results showed the importance of high index of inter-judge agreement when using subjective parameters of speech evaluation, especially when compared to an instrumental evaluation. This suggests the need for strategies for training and calibration of judges in the perceptual judgment to improve the reliability of auditory-perceptual assessment.
Hospital for Rehabilitation of Craniofacial Anomalies, Universidade de São Paulo - USP, Bauru, SP, Brasil.
Rafaeli Higa
RH
Scarmagnani
Hospital for Rehabilitation of Craniofacial Anomalies, Universidade de São Paulo - USP, Bauru, SP, Brasil.
Adriana Cristina de Almeida Santos Furlan de
AC
Oliveira
Hospital for Rehabilitation of Craniofacial Anomalies, Universidade de São Paulo - USP, Bauru, SP, Brasil.
Ana Paula
AP
Fukushiro
Universidade Estadual Paulista "Júlio de Mesquita Filho" - UNESP, Bauru, SP, Brasil.
Manoel Henrique
MH
Salgado
Hospital for Rehabilitation of Craniofacial Anomalies, Universidade de São Paulo - USP, Bauru, SP, Brasil.
Inge Elly Kiemle
IE
Trindade
Hospital for Rehabilitation of Craniofacial Anomalies, Universidade de São Paulo - USP, Bauru, SP, Brasil.
Renata Paciello
RP
Yamashita
D002972:Cleft Palate; D006801:Humans; D016015:Logistic Models; D015588:Observer Variation; D015203:Reproducibility of Results; D013068:Speech Production Measurement; D014681:Velopharyngeal Insufficiency / Q000175:diagnosis*; D014832:Voice Disorders / Q000175:diagnosis*; D014833:Voice Quality