EyeEcho: Continuous and Low-power Facial Expression Tracking on Glasses

The most commonly used non-wearable technologies to track facial expressions are based on cameras placed in front of the user to capture the face. Researchers rely on the images captured by RGB cameras (Hsieh et al., 2015; Thies et al., 2015), thermal infrared cameras (He et al., 2013), and/or depth cameras (Hsieh et al., 2015; Ijjina and Mohan, 2014; Thies et al., 2015), or images from existing datasets (Kahou et al., 2013; Liu et al., 2014b, a; Rifai et al., 2012; Wu et al., 2018; Sebe et al., 2007) to develop algorithms to track subjects’ facial expressions. Recently, learning-based algorithms show impressive performance on tracking facial expressions with the support of fast-developing deep learning models, such as Convolutional Neural Network (CNN) (Ijjina and Mohan, 2014; Kahou et al., 2013; Rifai et al., 2012; Wu et al., 2018), Generative Adversarial Network (GAN) (Lai and Lai, 2018), Deep Belief Network (DBN) (He et al., 2013; Kahou et al., 2013; Liu et al., 2014a; Ranzato et al., 2011), etc. Due to their impressive performance of tracking facial expressions and minimum requirement of the amount of training data needed, frontal-camera-based algorithms have been used as the ground truth acquisition methods in many wearable facial expression tracking systems, with the help of several public libraries, e.g., the Dlib library (King, 2009) and the Apple’s ARKit API (Inc., 2022). Despite their satisfactory tracking performance, these technologies usually require capturing the full face in the image, which does not work well when the user’s face is occluded. Some researchers have put efforts in reconstructing users’ facial expressions when part of their face is occluded (Hsieh et al., 2015; Lai and Lai, 2018; Wu et al., 2018). However, the frontal-camera-based algorithms are still easily impacted by lighting conditions in the environment and does not work well while users are in motion.

Apart from frontal camera based methods, recently some researchers placed speakers and microphones in front of the user to recognize facial expressions and emotional gestures (Gao et al., 2022). This alleviated some privacy concerns brought by camera-based methods, but the system can only recognize discrete expressions and emotions. In the meantime, other researchers also use frontal speakers and microphones to detect more subtle facial movements of the user, which are blinking (Liu et al., 2021). mm3DFace (Xie et al., 2023) reconstructed users’ facial expressions continuously with a competitive performance by placing a millimeter wave (mmWave) radar in front of them. These non-camera-based technologies demonstrate the potential of sensing modalities other than cameras for tracking facial movements but still suffer from the limitations of non-wearable devices that they do not work well when the user’s face is occluded or the user is walking around.

In order to position our work better in the previous literature, we would like to provide a precise definition of continuous facial expression tracking. Generally, there are two types of systems for monitoring facial expressions, determined by whether the task they aim to solve is classification task or regression task:

The first category focuses on recognizing a set of pre-defined discrete facial expressions or gestures. Most prior wearable sensing systems (Xie et al., 2021; Verma et al., 2021; Ando et al., 2017; Iravantchi et al., 2019) fall into this category. They perform classification tasks and report the accuracy of distinguishing these facial expressions or gestures as performance metrics. However, the output of these systems does not provide information on how the face appears during the process of making a facial expression. This information is needed for downstream applications such as adding facial expressions to render a personalized avatar or enabling video conferencing on smart glasses without the need of holding a camera in front of the face.

The second category aims to estimate the position and shapes of all parts of the face (such as the nose, eyes, eyebrows, cheeks, and mouth) continuously, often multiple times per second. Technologies in this category usually carry on regression tasks. Most camera-based methods have been able to track facial expressions continuously. However, achieving continuous tracking with wearable sensing systems has been challenging, as it requires high-quality and reliable information about facial movements. Recently, several wearable methods have started exploring continuous estimation of facial expressions (Chen et al., 2020, 2021; Wu et al., 2021; Li et al., 2022; Kim and Harrison, 2023; Zhang et al., 2023c). As a starting point, they define a set of facial expressions for participants to perform. Unlike the methods in the first category, these systems are able to continuously provide the position and shapes of different facial parts as a user makes a facial expression, ranging from a neutral face to the most extreme state. The position and shapes of the facial expressions are represented using landmark parameters (Chen et al., 2020; Wu et al., 2021; Kim and Harrison, 2023; Zhang et al., 2023c) or blendshape parameters (Chen et al., 2021; Li et al., 2022), which are the output of these sensing systems. The performance is measured using the Mean Absolute Error (MAE) between the predicted parameters and the ground truth captured by a frontal camera. EyeEcho belongs to the second category, as it tracks facial expressions continuously using wearables. In Subsection 8.8, we further discuss various potential applications that can be enabled by the ability to continuously track facial expressions on glasses.

Prior work (Li et al., 2022; Wu et al., 2021; Zhang et al., 2023c; Chen et al., 2020) have demonstrated that partial skin and muscle deformations behind and inside ears are highly informative to reconstruct full facial expressions when they are captured by different kinds of sensors. Xie et al. (Xie et al., 2021) proved that the skin and muscle deformations around the eyes and the cheeks contain information that can be extracted to recognize upper facial gestures. The sensing hypothesis of EyeEcho is that these deformations around eyes and cheeks are highly informative about detailed facial movements on both lower and upper face including eyes, eyebrows, cheeks, and mouth. Considering that people’s facial skin and muscles are interconnected, moving any part of the face would inevitably stretch the muscles and skin on the entire face. EyeEcho applies this skin-deformation-based sensing hypothesis on glasses to develop research questions: is it possible to infer full facial expressions by only observing skin deformations around glasses (e.g., cheeks)? If so, what is the appropriate hardware set up including the number, orientation and position of the sensors? How well can it track facial movements on different areas of the face (e.g., blinking) under different real-world scenarios? To explore these research questions, we developed EyeEcho using active acoustic sensing to capture the skin deformations on glasses, which we will detail later. We chose acoustic sensing because its sensors are small, lightweight, low-power and have been successfully applied to various tasks on tracking human activities, including health-related activities detection (Nandakumar et al., 2015; Wang et al., 2018), novel interaction methods (Aumi et al., 2013; Wang et al., 2016; Yun et al., 2017; Xu et al., 2020), silent speech recognition (Zhang et al., 2020, 2023b, 2023a; Sun et al., 2023), authentication (Lu et al., 2019; Gao et al., 2019; Wang et al., 2022; Huh et al., 2023; Iwakiri and Murao, 2023), discrete facial expression recognition (Xie et al., 2021), finger tracking (Sun et al., 2018; Nandakumar et al., 2016), hand gesture recognition (Zhang et al., 2018b), body pose estimation (Mahmud et al., 2023), and motion tracking (Zhang et al., 2018a; Lian et al., 2021).

With the prototype of glasses completed, we then introduce how we adopt FMCW-based acoustic sensing to sense the skin deformations on glasses. The system overview of EyeEcho is displayed in Fig. 3.

In order to infer full facial expressions continuously from the skin deformations represented by differential echo profiles, we adopted a customized deep learning pipeline.

As discussed in Sec. 4.3.1, a full facial expression is represented by 52 blendshape parameters in our system. We evaluate the performance using the Mean Absolute Error (MAE) of the 52 parameters between the prediction of EyeEcho and the ground truth. Employing this common metric allows us to compare EyeEcho’s performance with that of prior work. It is important to note that the same MAE value can yield substantially different visualization results for the lower face and upper face in terms of how closely the predicted facial expression matches the ground truth, as perceived by human eyes. In order to help readers better understand the true performance of our system, we divided our evaluation metrics into two categories: 1) Lower-face MAE (LMAE) - evaluating 33 lower-face blendshape parameters related to the movements of cheeks, mouth, nose and tongue and 2) Upper-face MAE (UMAE) - evaluating the remaining 19 upper-face blendshape parameters related to the movements of eyes and eyebrows. Based on our observation and also from prior work (Li et al., 2022; Chen et al., 2021), there is little visual difference between the prediction and ground truth when LMAE is below 40 and UMAE is below 60. Therefore, we adopted two other evaluation metrics in the results, the Percentage of Frames with LMAE under 40 (PL40) and the Percentage of Frames with UMAE under 60 (PU60). In total, we report five metrics, including MAE, LMAE, UMAE, PL40 and PU60 in the following sections when reporting the tracking performance of EyeEcho.

We plotted the visualization of facial expressions with different MAE for three facial expressions, Open Mouth, Open Eyes, and Blink, in Fig. 7. The figure shows that for large facial movements such as Open Mouth (lower-face) and Open Eyes (upper-face), the prediction is visually similar to the ground truth when MAE is under 40. When MAE is around 20, the prediction is almost indistinguishable to the ground truth. For subtle facial movements, like blinking, the prediction is also highly similar to the ground truth visually when MAE is under 20.

We first analyze the performance of EyeEcho to track facial expressions continuously. With all the 12 remounting sessions of data collected under each scenario, we conducted a 6-fold cross-validation using 10 sessions to train the model and 2 sessions as the testing sessions to evaluate the results. We report the five evaluation metrics, as shown in Tab. 2.

Since our EyeEcho system tracks facial expressions continuously rather than classifying them, we capture the entire process of a facial expression transitioning from a neutral face to its most extreme state. In the previous subsection, we validated the overall performance of our EyeEcho system. However, it is also important to assess its performance in tracking different degrees of deformation for these facial expressions. Based on our analysis in Sec. 5.1.2, we calculated the degree of deformation for each frame and categorized all frames in the user study into four groups based on their degrees of deformation. We present the evaluation results in terms of average MAE for each category in Tab. 3, along with the percentage that each category represents among all the frames. As shown in the table, the majority of frames in our user study have deformation levels below 150. Our system demonstrates satisfactory performance on tracking facial expressions for these frames, as Subsection 5.2 establishes that the prediction and ground truth are visually highly similar when the MAE is below 40. For frames with deformation levels exceeding 150, the performance of our system is slightly lower, but it still maintains an acceptable level of accuracy, as depicted in Fig. 9. This analysis demonstrates that our system performs well across varying degrees of deformation for different facial expressions.

In the previous experiments, we used the data from 10 sessions (20 minutes of data) to train the model. However, 20 mins of training may not be always preferable for users in real-world deployments. Therefore, we further explore how much training data is needed before the system reaches an acceptable accuracy. We chose two sessions of data as the testing sessions and employed 2, 4, 6, 8 and 10 sessions of data respectively to train the model for each scenario. We showed the evaluation results in Fig. 10 (a). As we can see in the figure, the average performance of the system improved with more training data. However, even with just 2 training sessions (4 minutes of training data), the system achieved a MAE of 29.7 and 34.3 for the two scenarios. According to the analysis in Sec. 5.2 and Sec. 5.3.2, this result is already good enough to provide an acceptable tracking performance on facial expressions that are highly similar to the ground truth visually. In essence, two sessions of training data are likely enough to provide satisfactory tracking performance in real-world deployments for the majority of users. More training sessions could be collected for users who do not have good enough performance.

In the previous experiments, we employed a user-dependent model to predict facial expressions for each individual participant. This model was trained using the data specific to each participant. To investigate the degree of user dependency within the system, we conducted a Leave-One-Participant-Out (LOPO) experiment for each scenario. In this experiment, we utilized the data from 11 participants to train the model and then evaluated the results on the data from the remaining participant. This process was repeated for each participant, and an average result was obtained using this user-independent model.

The results of this user-independent model are presented in Figure 10 (b). Notably, in comparison to the results of the user-dependent model, the results were worse, with a MAE of 49.0 and 53.2 for the sitting and walking scenarios, respectively. However, this outcome was anticipated because our system relies on the reflection of acoustic signals on the face and head, which varies significantly among participants. Additionally, differences in how participants wore the device and executed facial expressions also contributed to this variance.

Subsequently, we explored a user-adaptive model, where we used a small portion of data collected from this participant (2 sessions) to fine-tune this LOPO model. As shown in the figure, the results for both scenarios significantly improved compared with the user-independent model, with an MAE of 25.7 and 30.5. Besides, the results were also better than those trained using a user-dependent model with the same amount of training data (2 sessions), with an MAE of 29.7 and 34.3. This result showed the potential for further performance enhancement. If the model can be trained with a much larger data set from more participants in the future, the performance of the model can be further improved with minimal training data from a new user.

All the experiments and analysis above separated the sitting and walking scenarios and reported the performance independently. This indicates a new user is required to provide data for both sitting and walking scenarios, which may not offer the optimal user experience. In this experiment, we explored using the data collected from the sitting scenario to train a model, which is transferred and evaluated on the testing data collected in the walking scenario. For this experiment, we compared three models. Among the three types of models, the testing data was the same two sessions collected in the walking scenario. The results were averaged across all 12 participants.

The first model is User-dependent Model (NT10), where 10 sessions from the walking scenario were used to train the model with no transfer learning applied. The average MAE for this model is 26.9. In the second model (Transfer learning Model (NT2)), the model was trained with 10 sessions of data from the sitting scenario and fine-tuned using 2 sessions of data from the walking scenario. This model yields an average MAE of 29.0. In the third model, only 2 sessions from the walking scenario were used to train the model. The average MAE in this case is 34.3. The results show that transfer learning improves the MAE from 34.3 to 29.0, using the same size of training sessions from the walking scenario. It shows that EyeEcho has the potential to be adapted to the new walking scenario with minimal training data needed. We plan to explore how to use advanced transfer learning to further reduce the training data for new scenarios in the future.

The EyeEcho system was designed to operate within the frequency range of $16-20kHz$, with the goal of easy adoption on most commodity speakers and microphones since the acoustic sensors in most commodity devices can sample up to 48KHz. In post-study surveys, participants did not report any issues with hearing the acoustic signals. While it is possible that some users may be able to hear it, EyeEcho can readily adapt to higher inaudible frequencies with minimal impact on tracking performance. In order to validate this assumption, we shifted the operating frequency range of the EyeEcho system to $20-24kHz$ and conducted a new in-lab user study of the same procedure described in Sec. 5.1 with 10 participants (3 females and 7 males, 22 years old on average). With the 12 sessions of data we collected from each participant, we also ran a 6-fold cross-validation by using 10 sessions to train the model and 2 sessions to evaluate the performance. The average MAE of each participant and all 10 participants on average are demonstrated in Fig. 11.

As shown in Fig. 11, the average MAE of all 10 participants in this study is 23.6, which is comparable to the MAE of 22.9 in the first study in Sec. 5 with the operating frequency range set at $16-20kHz$. This validates that EyeEcho can be adapted to a higher inaudible frequency range ($20-24kHz$) with little impact on the system performance. Furthermore, we specifically asked each participant "Can you hear the sound emitted from our system? Yes / No" in the questionnaires collected at the end of this study and all 10 participants answered ’No’ to this question. Therefore, we believe that EyeEcho can operate at a frequency range that has minimal impact on users’ daily activities with a satisfactory tracking performance.

Since EyeEcho uses acoustic signals as the sensing method, there is a chance that everyday environmental noise could have a negative impact on the tracking performance. To investigate how different environmental noises can affect performance, we further extended the new study in Sec. 6.1. The study was originally conducted in a quiet meeting room with the background noise of the air conditioner, as shown in Fig. 12 (a). To evaluate the EyeEcho system under different noisy environments, after the first part study was completed in the quiet room, we then asked the participants to move to different environments to collect testing data with the existence of various types of noises: (1) Music Noise: in the experiment room with random music played (Fig. 12 (a)); (2) Cafe Noise: in a cafe with cafe staff and customers talking (Fig. 12 (b)); (3) Street Noise: on the street near a crossroad with vehicles and pedestrians passing by (Fig. 12 (c)). In every one of the three noisy environments above, each participant performed facial expressions for 2 sessions (4 minutes) as testing data.

Then, we used the first 10 of 12 sessions of data collected in the quiet room to train a model, which was tested using the remaining 2 sessions of data in the quiet room and 2 sessions of testing data collected in different noisy environments. Please note that no data collected in the noisy environments was used for training. The evaluation results are displayed in Tab. 4. We also measured the noise level in each noisy environment for each participant and showed the average measurement in Tab. 4.

The evaluation results in Tab. 4 demonstrate that the average MAE for 10 participants remains consistent at 27.3 and 28.7 with the presence of music noise and cafe noise compared with the MAE of 24.2 in the quiet environment. There is a small performance variance for some participants among these three environments because testing data is not large-scale but overall the system is resistant to these two noises considering that the prediction is visually similar to the ground truth when the MAE is below 40 as discussed in Sec. 5.2. However, the system performance dropped significantly when the user study was conducted on the street. We further explore the possible causes below.

We first plotted both the signal with noises and the pure noises in the frequency domain in Fig. 13. As we can see in the figure, all three noises are mostly within the audible frequency ranges and will be filtered out by the band-pass filter in our system. Besides, the strength of the noises is much smaller than the signal because the sources of these noises are relatively far away from the microphones in our system. This helps explain why the music noise and the cafe noise have little impact on the system performance. In theory, street noise should also have a limited impact on the system’s performance. However, the evaluation results suggest otherwise. Hence, we further analyzed the received acoustic signal in the frequency domain in different noisy environments. We used P5’s data as an example for illustration and plotted the signals under different noisy environments since the signal patterns are similar among all participants. As shown in Fig. 14, the signals are very similar to each other in the frequency domain under the first three environments while the signal looks quite different when collected on the street. Fig. 13 already shows that the noises have a limited impact on the signal so we believe this difference is mainly caused by the temperature difference between indoor and outdoor environments. According to prior research (Hayashida et al., 2020; Rayes, 2022), the frequency response of both speakers and microphones can be largely impacted by the temperature of the environment where they operate. The datasheets of the speakers (Knowles, 2022) and the microphones (InvenSense, 2022) used in our system suggest that the speakers can be more vulnerable to the temperature change. Our study was conducted in a cold region where the outdoor temperature varied from $-5\degree C$ to $5\degree C$ when the testing data on the street was collected for 10 participants while the indoor room temperature remained between $20\degree C$ to $25\degree C$.

To further verify this hypothesis, we did one experiment: we moved the EyeEcho system from the indoor environment ($22\degree C$) to the outdoor environment ($1\degree C$) and recorded the received signal with our system at different times after the system was brought outdoors. The change of the received signal in the frequency domain is presented in Fig. 15. As the figure shows, the received signal in the EyeEcho system visually changed after the temperature decreased. Since only the on-street testing data was collected outdoors, this change only affected the system performance for the testing data collected on the street, because the training data and testing data are significantly different in this condition.

In summary, our EyeEcho system is robust to different kinds of daily noises because it operates in the ultrasonic band. However, our system may need further calibration in the areas where the temperature is significantly low. This can be achieved by choosing sensors whose frequency response is more resistant to temperature changes and collecting more training data under different temperatures. We will explore this in the future.

To explore the usability of the EyeEcho system, participants were requested to finish a questionnaire at the end of the study in Sec. 6.1. The participants first rated their overall experience with EyeEcho by answering two questions: (1) "How comfortable is this wearable device to wear around the face? (0 most uncomfortable, 5 most comfortable)"; (2) "How acceptable do you find the weight of our wearable device? (0 most unacceptable, 5 most acceptable)". On average, 10 participants gave scores of 4.2 and 4.8 to the two questions above. All 10 participants agreed with the statement "The pair of glasses is easy to use." except that P9 reported that "glasses would fall in a squeeze gesture". As for the question "Compared with normal glasses, what do you think of this device?", 9 participants thought that EyeEcho is generally very similar to a pair of normal glasses while P1 thought it is a little bit more hard to be put on than normal glasses because the legs of the glasses cannot be bent. Meanwhile, P3 and P10 suggested that EyeEcho could have selected prescription lenses for different users in future while P4 believed that it will be easier to wear EyeEcho if all sensors are completely embedded into the legs of the glasses. We believe that all these suggestions are valuable and we will take them into consideration in future improvement of the prototype design.

At last, we asked participants that "If there is a product like this to track your facial expressions in future, do you want to use it?". 7 participants answered ’Yes’ to this question while 2 participants answered ’Maybe". P8’s choice is dependent on the performance of the system and he "may consider this device if it can perform nearly as well as existing technology".

Evaluating a facial expression tracking system in real-world environments presents significant challenges, primarily due to the absence of a suitable method for acquiring ground truth data that users can comfortably wear during their daily activities. For instance, prior research, as well as our own study for both sitting and walking scenarios in the lab, have relied on placing a camera in front of the users’ face to capture ground truth data of facial expressions. However, it becomes nearly impractical to expect participants to wear a ground truth acquisition device in a completely uncontrolled environment, where they have the freedom to go to any location (especially outdoor locations) and engage in any activity (especially activities such as driving) without the presence of researchers.

The lack of reliable and minimally-obtrusive wearable systems that can track users’ facial expressions continuously has also been a significant motivator behind the development of EyeEcho. In our initial study, we successfully showcased the promising performance of EyeEcho in a controlled lab setting, where we simulated various real-world scenarios, including factors like motion and noise. Based on the exciting results, we also would like to demonstrate that our proposed system can perform effectively in a more naturalistic setting since our core sensing principle, which involves tracking inaudible acoustic reflections on the face, is less susceptible to environmental influences.

To validate our hypothesis, we designed and conducted a second study, referred to as a semi-in-the-wild user study in which participants engaged in various daily activities within a more naturalistic setting—a one-bedroom apartment. In this study, we aim to evaluate EyeEcho in an environment as natural as the real-world setting, while ensuring that the ground-truth acquisition system works well. To the best of our knowledge, this study marks the first attempt in the field to evaluate a non-camera-based wearable facial expression tracking system in a more naturalistic real-world setting instead of controlled lab settings.

The primary goals of this study are as follows:

• •

  Evaluate the performance of EyeEcho in multiple rooms within a home environment, where furniture and layouts vary;

• •

  Assess how well EyeEcho can track natural facial expressions that occur during various daily activities.

The study was conducted in a one-bedroom apartment of a researcher off campus, with three rooms: the living room, bedroom, and kitchen, as depicted in Fig. 16. The participants were instructed to perform different activities in a random order while wearing the experimental devices within these rooms, as follows:

• •

  Living room: Watching videos on a computer, reading, describing things, having conversation with the researcher while walking around;

• •

  Bedroom: Watching videos on a computer, reading, describing things, having conversation with the researcher while making the bed;

• •

  Kitchen: Watching videos on a computer, describing things, having conversation with the researcher while washing the dishes and using the microwave oven.

These activities were intentionally designed to elicit a range of natural facial expressions and movements that happen in everyday life. For example, the videos that participants watched included online videos categorized to specifically evoke different emotional experiences¹¹1https://www.alancowen.com/video and pre-selected YouTube videos consisting of various funny/scary scenes that happened in a movie or in real life. Watching these videos led to spontaneous and varied facial expressions. While reading, describing things and having conversations, participants had facial movements frequently. According to the standard defined in Sec. 5.1.2, among all the frames collected in this user study, there are 84.4% and 24.6% frames in which participants deformed their face with a degree of deformation over 50 and 100, respectively. This is comparable to the degrees of deformation that the participants performed in the in-lab study.

For this semi-in-the-wild study, we recruited 10 participants (8 female, 2 male) with an average age of 23 years. Each participant received USD $20 compensation for each study day. The study was conducted in the one-bedroom apartment as detailed above.

Throughout the study, the participants wore the glasses embedded with the EyeEcho system and a chest mount, as used in the walking study, to facilitate ground truth capture via an iPhone placed in front of them. The chest mount was tested for comfort and usability before the study. Each participant completed the study over two days with a gap less than one week, engaging in various activities as described earlier. On the first day, participants conducted a 10-minute training session in the living room, followed by 10-minute testing sessions in all three rooms. The order of the rooms was randomized. On the second day, participants completed 10-minute testing sessions in all three rooms, with no additional training data collected. In total, each participant contributed approximately 70 minutes of data (10 minutes training on Day One, 60 minutes testing on both days). The entire study duration for each participant did not exceed 2.5 hours.

In the study, a researcher remained outside the experiment room to provide instructions and engage in conversations with participants via smartphone, simulating real-world scenarios where users may have video conferences while multitasking or moving around without a camera continuously in front of them.

We used a current ranger and a multimeter to measure the current and voltage of our EyeEcho system while all components were in operation. The measurements showed that the current flowing through our system was $41.1mA$ at the voltage of $4.07V$. Therefore, the power consumption of EyeEcho is around $167mW$. This power consumption should allow EyeEcho to work on current smart glasses or AR glasses for a reasonable period of time.

Smart glasses often come with limited battery size due to their compact device size. For instance, Amazon Echo Frame with 4 speakers can last about 2 hours with audio on (or its affiliates, 2022). The Ray-ban Stories Smart Glasses have a battery capacity of $167mAh$ (Scott-Thomas, 2023). If EyeEcho is deployed on it and used alone, the glasses can last 4 hours in theory. On the other hand, AR glasses often come with a larger size and battery life. For instance, the battery capacity of Google Glass, Espon Moverio, and Microsoft HoloLens are $570mAh$ (Wikipedia, 2023), $3400mAh$ (Epson, 2022) and $16500mAh$ (Niora, 2022), guaranteeing around 14, 83, and 402 hours of battery life in theory, if EyeEcho is used alone.

As stated above, our power consumption is already relatively low, especially compared to using cameras for facial expression tracking. However, what we present in this paper is just a starting point. The power consumption of our system can be further optimized in the future. For instance, our measurement indicates that the two speakers take up about 80% ($135mW$) of the system’s power consumption ($167mW$). Therefore, depending on applications, EyeEcho, especially the speakers, do not need to be turned on all the time. Besides, reducing the loudness of the speakers and/or using high-efficiency speakers can also lead to lower power consumption. For instance, we replaced the two speakers in our system (SR6438NWS-000) with two speakers that are more power-efficient, OWR-05049T-38D (BOGO, 2023) as shown in Fig. 19, and adjusted the Sound Pressure Level (SPL) to the same value as the previous speakers. Then we measured the current flowing through the system again and got the value $17.3mA$ with the new speakers, which gave us a power consumption of $71mW$. This validated that EyeEcho’s power can be further reduced by adopting more power-efficient speakers.

To demonstrate how EyeEcho can be integrated with commercial devices, we deployed the data processing and deep learning pipeline of EyeEcho with the help of PyTorch Mobile (AI, 2022) on an Android smartphone (Xiaomi Redmi K40, Android 12, Qualcomm Snapdragon 870 SoC). Keeping the algorithms deployed on the smartphone will eliminate the need to transmit the data to a server on the cloud, which can better preserve the privacy of the user.

In order to keep the algorithm making predictions fast enough in real-time, we replaced the deep learning model with a ResNet-18 architecture. To demonstrate that this lighter model results in a similar performance, we trained the lighter model (ResNet-18) in the same way as we did in Sec. 5.3 for the sitting scenario, achieving comparable performance with the full model (23.1 vs 22.9). For the usage of the real-time pipeline, we first trained the deep learning model with the data collected from users in PyTorch on a NVIDIA GeForce RTX 2080 Ti GPU. Then we traced the trained model to make it applicable to deployment on smartphones. The traced model was loaded onto the Android phone (Xiaomi Redmi K40) and used in the Android application we developed for data collection and facial expression prediction. During the inference stage, the running App continuously received data streamed from the BLE module in our EyeEcho system via Bluetooth, preprocessed the received data (i.e. organizing data based on channels, filtering the raw data, and calculating the echo frames), and fed echo profiles into the traced deep learning model for predictions. The predicted blendshape parameters representing users’ facial expressions were finally streamed from the smartphone to a laptop via Wi-Fi for downstream applications. On average, it took $34ms$ to make one inference, which led to a refresh rate of 29 FPS. We believe that this is sufficient for most applications since most videos can be played at 30 FPS. With the system implemented on the smartphone, we were able to predict users’ facial expressions in real-time and rendered them with a personalized avatar powered by Avaturn²²2https://avaturn.me/, as shown in Fig. 20.

Blinking is an important part of facial movements and can be used to monitor health conditions and help diagnose many eye diseases of users (Park and Baek, 2019; Dementyev and Holz, 2017). In the past, in order to detect blinks, separate sensors have been installed on glasses, such as cameras (Xiong et al., 2017) and capacitive sensor (Liu et al., 2022). To be best of our knowledge, there is no acoustic-based system that can detect blinks based on skin deformations around the cheeks.

In our preliminary experiments, we noticed that blinks also lead to substantial deformations on tissue and skin around the face. This was visually evident in the echo profiles extracted from the received acoustic data. Consequently, we believe that, in addition to tracking facial expressions, our system has the capacity to detect blinks.

To evaluate the feasibility of EyeEcho for blink detection, we conducted a follow-up study with 6 participants. In this follow-up study, we asked each participant to watch some video clips of landscapes while wearing the EyeEcho system. We did not instruct them to blink intentionally. Instead, EyeEcho system detected the natural blinks of participants while watching these videos. While wearing the EyeEcho device, they watched five 2-min video clips. Between two videos, they remounted the device. For the five sessions of data, we ran a 5-fold cross-validation using 4 sessions to train the model and testing on the remaining session. During training, we only used 2 out of 52 blendshape parameters that are related to blinking (eyeBlink_L/R) as the ground truth for the model to optimize blinking detection performance. The average F1 score of blinking detection across 6 participants was 82% ($std=12\%$) for our system. It’s important to note that EyeEcho was not specifically designed for blink detection, so performance could be further enhanced by incorporating additional sensors positioned closer to the eyes on glasses and oriented directly towards the eyes. This study aims to showcase the potential feasibility of using EyeEcho to simultaneously track facial expressions and blinks.

Over time, the prediction of the system could become worse because the user’s body status is changing every day and will not be exactly the same as the day when the training data is collected. In the semi-in-the-wild study in Sec. 7, we validated that our system still works well on Day Two when training data is collected on Day One. In this subsection, we would like to conduct a more thorough long-term evaluation of the system. Thus two researchers collected the same amount of training data as we did in the in-lab user study and tested the performance of the system on the same day as well as 1 day, 2 days, 1 week and 2 weeks after the training data was collected. The average testing results are shown in Tab. 5. The average MAE across the two researchers are 19.9, 33.6, 40.4, 36.3 and 36.0 for the test data collected on the same day and 1 day, 2 days, 1 week and 2 weeks after the training data was collected. The result shows that the performance of our system decreases over time but is still good enough even after 2 weeks, based on analysis in Sec. 5.2 and Sec. 5.3.2, which proves the stability of our EyeEcho system.

In Sec. 6.1, we validated that EyeEcho can operate in the frequency range $20-24kHz$, which is inaudible to users. However, even though the users cannot hear the signal, it may still cause health concerns to them. Thus, we used the NIOSH Sound Level Meter App³³3https://www.cdc.gov/niosh/topics/noise/app.html to measure the signal level of the EyeEcho system. We kept the speakers in the system emitting the signals and attached the microphones of the phone with the NIOSH app running directly onto the speakers. The average sound level measured was 48.2 dB. When we moved the microphones of the phone to the same distance from the speakers of EyeEcho as where the users’ ears will be if they wear the system, the measurement of the sound level was 37.8 dB. According to Howard et al. (Howard et al., 2005), the recommended ultrasound exposure limit for frequency around $20kHz$ is 75 dB. Therefore, we believe that our EyeEcho system is safe to wear for long-term use since the sound level is far from the recommended limit.

EyeEcho can preserve privacy by avoiding the use of cameras and limiting the frequency range of the sensing system. EyeEcho uses a band-pass filter to remove all frequencies other than $16-20kHz$ in the received signal, which means that all audible sounds including heavy privacy information (e.g., human speaking, environmental sounds) are removed. Furthermore, we also demonstrated that EyeEcho can conduct all computations locally on a smartphone without sending any data over the Internet in Sec. 8.2. In this way, sensitive information stays confidential because only the predicted facial expressions represented by 52 parameters instead of the original signals captured by microphones are shared with others. With future advancement in low-power on-chip deep learning, it is also possible to deploy everything on a single chip, further eliminating the privacy and security risks.

Enabling facial expression tracking on glasses have a wide range of applications, from enhancing video conferencing experience to novel input methods. Video conferencing, a common mode of communication, can be significantly improved using our system. Currently, during online video conferences, participants must position themselves in front of a camera or hold it to ensure others can see their facial expressions. However, with EyeEcho integrated into smart glasses, users can engage in real-time video conferences and convey facial expressions effortlessly, even while walking or multitasking. We have implemented a system capable of generating personalized avatars with facial expressions for each user, as detailed in Sec. 8.2. Consequently, the user experience in video conferencing with EyeEcho resembles traditional camera-based methods, but with the added convenience of hands-free operation.

Facial expressions can also serve as a novel input method, a concept has been explored in previous research efforts (Rantanen et al., 2010; Lankes et al., 2008; Matthies et al., 2017; Mei Yin et al., 2018). Our glasses-based system is well-suited for implementing this functionality, allowing different facial expressions to serve as distinct commands for interacting with smart or augmented reality glasses.

Moreover, facial expressions are linked to various health conditions. For instance, individuals with Parkinson’s disease may experience a loss of facial expressions. EyeEcho can be instrumental in tracking and monitoring the symptoms of such diseases, potentially contributing to improved healthcare outcomes.

Despite the promising performance, EyeEcho also has limitations that need further investigation.