Manual vs. automated annotation of non-manual signs: ELAN vs. FaceReader

Abstract

For more than a decade, signed language linguists have employed the free software ELAN to annotate manual and non-manual signs (NMSs) (Wittenburg et al. 2006). In the past ten years, several studies have sought to advance the automatic recognition of NMSs (Antonakos, Roussos, & Zafeiriou 2015; Bhuvan et al. 2016), including Hanada (2023), who proposes using the paid software FaceReader for the automated recognition of facial expressions and head movements. The present study aims to investigate whether manual transcription of NMSs in ELAN is consistent with automated transcription in FaceReader—that is, whether both transcriptions are equivalent and reliable. To this end, the same signed data from Hanada (2024) were used to produce both transcriptions, with ELAN labels matched to the Action Units (AUs) and head positions coded by FaceReader. In total, 649 NMSs were manually transcribed in ELAN, and 2,240 were automatically coded in FaceReader. The degree of agreement and disagreement regarding whether a given NMS was marked as active or inactive was then analyzed. The results show that the two tools agreed in 48.03% of the data. The main discrepancies were due to: (1) variations in the baseline used to define the participant’s neutral face in the human annotations, which depended on what the annotator had recently perceived as neutral during their analysis; (2) the fact that when two or more movements co-occur, only the most salient or intense one is annotated in ELAN; and (3) the face that FaceReader is not designed for linguistic analysis and annotates all detected facial movements, including those that may not have a linguistic function, while also assigning a degree of deviation to head positions even when the head is at rest. Therefore, it is not recommended to rely solely on FaceReader for analyzing NMSs. However, it can serve as a second annotator to support intercoder reliability (ICR) in qualitative studies. The findings of this study highlight the advantages and limitations of each transcription method and contribute to a better understanding of which research contexts may benefit more from one type of annotation than the other. Future research could examine how often related muscles co-occur and how this affects the annotation of distinct NMS movements.