It has been well documented in numerous studies that hearing loss (HL) has negative impacts on various aspects of life, including communication, job and academic performance, and social interaction and isolation [1-6]. Since individuals with HL have difficulty hearing, visual cues, such as facial movements and gestures, play a vital role in communication. Several studies have demonstrated the benefits of audiovisual (AV) integration for this population [7-13]. Grant, et al. [7] reported better consonant and sentence recognition in AV than in auditory only (AO) or visual only (VO) conditions. In the study by Puschmann, et al. [12], adults with HL showed a significantly higher target word detection accuracy in AV compared to AO when listening to running speech in noise. This AV benefit was also observed in hearing aid (HA) users, with Moradi, et al. [11] demonstrating improved consonant and vowel recognition in HA users in AV than in AO. Given the findings that individuals with HL rely on visual information in various conditions, including HA use and noisy environments, it is crucial to incorporate visual information into assessment and rehabilitation tools for accurately evaluating and enhancing their communication abilities. However, current assessment and rehabilitation tools targeting individuals with HL mostly utilize auditory stimuli, which poses limitations in reflecting real-life conversational settings and AV benefit [14,15]. There are some tests that include background noise (i.e., babble) to mimic real-life environments, but they do not incorporate any visual cues. Besides, in most settings, these tests are conducted in a sound-treated booth with one or two speakers which may not be enough to fully reflect the real-world environment. From a rehabilitation perspective, such controlled test environments can lead to a gap between test results and perceived benefit. For instance, HA users still experience difficulty understanding speech in noise, such as restaurants and café [16-18], even when their devices are well-fitted by healthcare professionals. If this problem continues, individuals might think that they are unable to hear well even with HAs and this could lead to dissatisfaction towards the devices. Eventually, they may return or give up wearing the devices, resulting in low compliance with aural rehabilitation.

Virtual reality (VR), a key technology of the Fourth Industrial Revolution, offers immersive environments, meaning that individuals feel present and involved in the virtual space, through simultaneous presentation of auditory and visual information [19]. In the field of audiology, Stecker [20] noted that compared to the distracting environment of the current laboratory setting, which consists of a sound booth and a monitor, VR offers advantages, such as being more entertaining and engaging, as well as ensuring consistency in the presentation of visual and auditory information and enabling naturalistic multi-dimensional tasks. However, when examining the various areas where VR has been applied, most studies focus on sound localization and audiology training, while research evaluating speech performance using VR mostly targets children [21-26]. Salanger, et al. [26] assessed the effectiveness of employing VR technology in pediatric hearing research by comparing speech perception performance in a conventional laboratory setting and a simulated VR classroom setting. Children completed speech perception and sound localization tasks in AO and AV. The results showed higher speech perception performance in AV and their performance in the VR classroom setting resembled their performance in the conventional laboratory setting for the AV condition than for the AO condition.

The purpose of this study was to evaluate speech performance in various real-world communication scenarios using VR. Considering reliance on visual information and the challenges with speech understanding in noise even after wearing HAs, individuals with HL were divided into HA users and non-HA users. Additionally, to explore the potential of a VR testing, the relationship between VR testing method and conventional testing method was investigated.

In this study, individuals with HL were divided into nHA and HA users, and their speech performance was evaluated in AO, VO, and AV conditions in both quiet and noise. The study examined the relationship between speech performance and subjective listening ability as assessed by the K-SSQ, as well as the potential of the clinical applicability of VR testing method by investigating the relationship between conventional speech test performance and multimodal speech performance in AV. Findings of the study are consistent with prior research that integration of auditory and visual information improves speech understanding for individuals with HL [9,10,23,29,30]. For example, Lidestam, et al. [29] compared speech-in-noise performance between AO and AV by dividing participants into two groups (AO training and AV training), with stimuli (word and consonants) presented at 0 dB SNR, and reported that performance improved only in AV. In our study, both the nHA and HA groups showed the best speech performance in all test conditions (unaided, aided, quiet, and noise) in the order of AV, AO, and VO even when the speech testing was conducted using VR. This suggests that, regardless of whether HAs are worn or the presence of noise, the provision of visual information together with auditory information contributes to improved speech understanding. However, when examining modality and noise conditions in multimodal testing, different characteristics of speech performance was observed. For most conditions, speech performance was highest for the nHA group, followed by HA users in aided condition and HA users in unaided condition, suggesting that visual information and usage of HAs may compensate for HL, particularly in the real-world communication scenarios. In VO, HA users showed better performance in the unaided condition than in the aided condition, indicating that visual information may play a stronger role in the absence of auditory information. The statistical analysis using ANCOVA showed a significant main effect of modality, with no significant interaction between modality and condition, nor a significant main effect of condition. This suggests that the modality, whether auditory, visual, or AV had a stronger influence on speech performance than the presence of noise. A significant difference was observed between AO and VO through post-hoc comparisons which is consistent with previous studies that when it comes to speech perception, one modality (auditory in this case) could dominate while information from other modalities serves a supplemental role [31-34]. The sound-induced flash illusion task serves an example; when a single visual cue (flash) is presented alongside two auditory stimuli (beeps), individuals perceive two flashes. This indicates that audition could dominate vision which may not always be the dominant modality [34]. Overall, findings of the study emphasize the importance of considering both auditory and visual information when assessing speech performance in individuals with HL.

Our study is meaningful in that it utilized VR to assess speech performance in individuals with HL in a more ecologically valid setting, incorporating café noise and a variety of test conditions related to communicative environments. Compared to other studies, our study setup used more naturalistic stimuli and testing environments that could be easier to set up when employed in clinical practice. Most auditory research studies using VR used avatars or objects. For instance, Kepp, et al. [35] investigated pitch ranking abilities in children with cochlear implants, HAs and normal hearing using VR. The visual stimuli used in this study was a cockpit of a helicopter so that the participants felt like they were flying forward. Bakhos, et al. [21] explored the effectiveness of VR training for audiology students and an avatar was used as a virtual patient. The video recordings used in our study were created with actors speaking sentences and conversing at a café, with café noise incorporated for the noise condition.

Although the findings suggest the potential of VR-based multimodal testing in capturing AV and HA benefit, they also highlight important considerations when employing such testing in practice. First, while the presentation level of the speech material was controlled at 60 dBA—commonly regarded as conversational level—no adjustments were made to the HA settings in order to preserve the naturalistic communication environment. HA features, such as directionality and noise reduction could have reduced the noise more during testing in noise, contributing to the variability that could confound group differences and noise resilience comparisons. Second, to examine HA benefit within the multimodal condition, speech performance of HA users was evaluated under both unaided and aided conditions, but this resulted in a reduced sample size for each subgroup. Therefore, while findings of the study provide preliminary insight, future studies with larger sample sizes are needed to explore subgroup characteristics. Furthermore, the study found no significant correlations between multimodal speech performance with VR and either the K-SSQ scores or conventional speech testing. These findings may be attributed to several factors. The K-SSQ captures a broad range of subjective listening abilities across various environments, whereas the VR testing focused on a specific communication scenario. This mismatch in ecological coverage may have weakened the correlation. Individual differences in visual reliance and listening strategies may not also have been adequately assessed by either the K-SSQ or the conventional test. These considerations highlight the complexity of linking subjective and objective auditory measures, especially when ecological validity is prioritized. It is worth noting that there is also a need for unification of the definition and a setup for high ecological validity [36]. Keidser, et al. [36] mentioned that there has been a significant increase in studies using more realistic environments or tasks. However, there are still limitations that the definition of ecological validity is different across studies and there are many factors contributing to ecological validity and the underlying process of communication.

In sum, while our findings suggest that VR-based multimodal testing may enhance ecological validity by more closely resembling real-world communication environments, the current data do not provide clear evidence that this method is more effective than the conventional testing. In order to validate the feasibility of the clinical application of such testing, future studies with larger sample size, diverse participant characteristics (i.e., reliance on visual information, HA use etc.), speech materials matching the virtual space, speakers with various speech acoustic features (i.e., male, female, and child) would be necessary.