GazeTrak: Exploring Acoustic-based Eye Tracking on a Glass Frame

Webcams have been widely used to implement eye tracking technologies because of its ubiquity on computers and its advantage of low-cost. Researchers have done many work to explore the potential of webcams for eye tracking (Papoutsaki et al., 2017; Papoutsaki, 2015; Saxena et al., 2022; Wisiecka et al., 2022). Currently, there are plenty of webcam-based eye tracking platforms that are available online, such as RealEye.io (sp. z o.o., 2022), GazeRecorder (GazeRecorder, 2022), and WebGazer.js (Group, 2021).

These webcam-based eye tracking platforms provide affordable solutions for eye tracking with acceptable tracking performance for everyday users. However, the position of webcams are usually fixed and they have a relatively low resolution. Therefore, their performance can be more easily impacted by factors like lighting conditions, occlusions, camera orientations, etc.

In order to provide more accurate and reliable solutions to eye tracking and make them more applicable to assorted scenarios, researchers have put lots of efforts in implementing other non-wearable eye tracking technologies other than webcam-based systems, most of which are based on cameras with a higher resolution than webcams. Frontal camera-based eye tracking technologies based on computer vision techniques can take full advantage of the whole facial information of the user for eye tracking, which usually leads to high tracking accuracy. Different kinds of cameras have been used in tracking eye movements, such as RGB cameras (Ablavatski et al., 2020), infrared (IR) cameras (Ohno et al., 2003), and thermal cameras (Wang et al., 2016). Beyond using just one camera, many technologies adopted multiple cameras in their eye tracking systems in order to improve the performance in different perspectives including providing larger tracking coverage (Ahlstrom and Dukic, 2010), allowing for user motion (Hennessey and Fiset, 2012) and tracking eye movements of multiple users (Mahanama, 2022). Because of the reliable tracking performance and reasonable calibration time needed, frontal camera-based eye tracking technologies have been well commercialized, among which Tobii Pro Fusion (AB, 2022a) is one of the best desktop eye trackers because it only requires seconds of calibration process for new users and can provide a tracking accuracy as low as $0.3\degree$ in optimal conditions. As a result, this product has been used as reference in many research projects.

The aforementioned frontal camera-based technologies are mostly located at fixed positions and do not work well while users move to another position or are walking around. In order to allow some mobility for users while they are using the eye tracking technologies, many researchers investigated utilizing the cameras on mobile devices to track eye movements, such as mobile phones (Zhu et al., 2018; Lei, 2021; Krafka et al., 2016) or tablets (Bafna et al., 2021; Krafka et al., 2016). However, these eye tracking technologies based on mobile devices still require users to hold the mobile devices in front of their face all the time and cannot provide completely hands-free and motion-free experiences for users.

To overcome the challenges that non-wearable eye tracking technologies face as described in the last two subsections, many wearable eye tracking technologies based on cameras (Zhang et al., 2011; Ryan et al., 2008; Noris et al., 2011; Cho et al., 2012; Kassner et al., 2014; Lanata et al., 2015; Mayberry et al., 2014, 2015), optical sensors (Borsato and Morimoto, 2016; Topal et al., 2008b, a; Li and Zhou, 2018; Li et al., 2017), acoustic sensors (Golard and Talathi, 2021), magnetic sensors (Whitmire et al., 2016), Electrooculography (EOG) sensors (Bulling et al., 2008, 2009), or inertial measurement units (IMU) (Lanata et al., 2015) have been deployed on different kinds of wearables including glasses (Topal et al., 2008b, a; Zhang et al., 2011; Borsato and Morimoto, 2016; Ryan et al., 2008; Cho et al., 2012; Golard and Talathi, 2021; Li and Zhou, 2018; Mayberry et al., 2014, 2015), goggles (Bulling et al., 2008, 2009), hat (Lanata et al., 2015), and head-mounted devices (Whitmire et al., 2016; Noris et al., 2011; Kassner et al., 2014; Li et al., 2017). Among all these wearable eye tracking technologies, camera-based ones usually outperform others in terms of tracking performance and do not require lots of calibration data from new users. Many wearable eye trackers using cameras, especially on glasses, have become commercial and can be used as a reliable way to track eye movements continuously, such as Tobii Pro Glasses 3 (AB, 2022b), Pupil Labs (Invisible, Core, VR/AR add-ons) (Labs, 2022), Dikablis Glasses 3 (Ergoneers, 2022), and SMI Eye Tracking Glasses (iMotions, 2022). With these technologies, various novel gaze-based applications have been enabled, including detection of eye contacts (Ye et al., 2012), interaction with devices (Kytö et al., 2018; Matsubara et al., 2015; Paletta et al., 2014), and monitoring mental health (Vidal et al., 2012; Li et al., 2020).

Despite of the promising tracking performance, current solutions to wearable eye tracking systems still have some limitations. First of all, many eye tracking systems above can only recognize discrete gestures (Zhang et al., 2011; Topal et al., 2008a, b; Bulling et al., 2008, 2009), limiting their performance in applications that need continuous tracking of the eyes. Camera-based wearable eye trackers can provide high accuracy in continuous eye tracking, but cameras are usually power-hungry, which makes them relatively impractical while deployed in wearables that need to be worn in everyday settings. To address this issue, Mayberry et al. (Mayberry et al., 2014, 2015) proposed low-power solutions to tracking gaze positions with cameras on glasses while maintaining promising accuracies. Despite of the impressive performance, changing lighting conditions can still be a problem for these camera-based systems, as the performance became worse in an outdoor setting (Mayberry et al., 2015). Besides, commercial eye trackers are usually expensive and do not provide open-source software for users, preventing them from being easily accessed and adapted by general users.

Recently, Li et al. (Li and Zhou, 2018) proposed a low-cost and battery-free solution to continuous eye tracking using near infrared emitters and receivers on glasses. It achieves competitive performance but they stated in the paper that this system can be impacted by direct sunlight and glasses movement, i.e. the remounting of the glasses. Besides, this work tracks the position and size of the pupil so we cannot directly compare it to our system. After conversion, its tracking accuracy of gaze positions is smaller than $2\degree$ in angular error. Another system using similar technology from the same group (Li et al., 2017) tracks gaze positions with an accuracy of $6.3\degree$, worse than the tracking accuracy of our system at $4.9\degree$. Golard et al. (Golard and Talathi, 2021) conducted a modeling and empirical study to prove that ultrasound can provide a low-power, fast and light-insensitive alternative for camera-based eye tracking technologies. However, it was evaluated on a physical 3D model of a human eye and used time-of-flight estimated from acoustic signals and not clear how it can apply on a real user.

To the best of our knowledge, GazeTrak is the first wearable sensing technology based on active acoustic sensing that can track gaze points continuously. We summarized and compared GazeTrak with some aforementioned wearable and webcam-based eye tracking techniques that can continuously track gaze positions in Tab. 1. These techniques are those that are most related to our system. Please note that commercial eye tracking wearables (AB, 2022b; Labs, 2022; iMotions, 2022) usually have camera(s) recording the video of the environment as well so we can only roughly compare them to our device in terms of power and weight.

In order to capture the formation around eyeballs, we use FMCW-based acoustic sensing, which has been widely proven effective to estimate distance and movements from complex environments (Nandakumar et al., 2015; Wang et al., 2018).

In order to implement the FMCW-based active acoustic sensing technique mentioned in the section above, we chose Teensy 4.1 (PJRC, 2023b) as the micro-controller to provide reliable FMCW signal generation and receiving in multiple channels. We designed a PCB board to support two SGTL5000 chips which are the same as the one on the Teensy audio shield (PJRC, 2023a). With this customized PCB board plugged onto Teensy, it can support as many as 8 microphones and 2 speakers. We chose the speaker called OWR-05049T-38D (DigiKey, 2023) and the MEMS microphone called ICS-43434 (TDK, 2023) to support signal transmission and reception. We also built customized PCB boards for the speaker and the microphone to make them as small as possible. We used the Inter-IC Sound (I2S) buses on the Teensy 4.1 to transmit data between the Teensy 4.1 and the SGTL5000 chips, speakers and microphones. The collected data is stored in the SD card on Teensy 4.1. Fig. 3 (a)-(e) show these components.

We designed the first form factor using a commodity glass frame. We glued 1 speaker and 4 microphones to each inner side of a pair of light-weight glasses. The speakers and microphones are symmetrically placed on the glasses, as shown in Fig. 3 (f).

Based on the experience we learned during the iteration process, there are three key factors we took into consideration while designing the final form factor of GazeTrak: 1) Type of glass frame: We started designing the form factor with a large glass frame because we believe it has more room for us to place sensors. However, the larger the glass frame is, the easier it will be for the frame to touch the skin, blocking the signal transmission and reception. As a result, we finally picked a relatively small glass frame with a nose pad that can support the glass frame to a higher position. Besides, the light-weight glasses minimize the pressure attached on the user’s nose, making it more comfortable to wear; 2) Sensor position: The speakers and microphones on two sides are symmetric because we believe the movements of two eyes are usually synchronized. On each side, we place the speaker on the frame of the glasses next to the outer canthi because it is easier for the speakers to touch the skin if they are placed above the cheekbones or next to the eyebrows, considering their height. The microphones are scattered on the frame as far away from each other as possible to capture more information by receiving signals travelling in different paths. The sensors are attached as far away from the center of the lenses as possible in order to avoid blocking the view of the user; 3) Stability: We found that the stability of the device severely impacts the performance of our system especially when users need to remount the device frequently. The anti-slippery nose pad prevents the glasses from sliding down the user’s nose. Furthermore, we added two ear loops at the end of the legs of the glass frame. They greatly helps fix the glasses position from behind ears and improves the performance of the system. Finally, we made the form factor as shown in Fig. 3 (f).

The prototype above is suitable for initial testing and comparison of different configurations. However, once the design of the prototype is finalized, we aim to create a more compact and less obtrusive form factor that is suitable for everyday use by users. To achieve this, we have designed two PCB boards, each containing one speaker and four microphones onboard, which can be attached to one side of the glasses. We have also deployed the Teensy 4.1 and the PCB board with SGTL5000 chips directly onto one leg of the glasses. To connect the micro-controller and the customized PCB boards, we have used flexible printed circuit (FPC) cables. The system has an interface that allows it to be powered by a Li-Po battery. The compact prototype is shown in Fig. 3 (g), and Fig. 1 shows a user wearing the prototype. We believe that this prototype can be easily adapted and attached to different types of glasses.

We have measured the weight of the prototype, and it carries a total weight of 44.2 grams, including the glasses, Teensy 4.1, PCB boards, and the Li-Po battery. Compared to camera-based eye tracking glasses, our GazeTrak device is much lighter. For example, Tobii Pro Glasses 3 (AB, 2022b) weigh 76.5 grams for the glasses and 312 grams for the recording unit. Our device has a significant advantage over camera-based eye tracking glasses in terms of weight.

We first tested our system using the first prototype in Fig. 3 (f) with 12 participants. A user-dependent model was applied to train a separate model for each participant. Among 12 sessions we collected for each participant, we conducted a 6-fold cross validation to test the tracking performance of our system by using 10 sessions (33.3 minutes) of data for training and 2 sessions (6.7 minutes) of data for testing. Using the evaluation metrics defined in Subsubsec. 3.2.3, we calculated the mean gaze angular error (MGAE) in degree for all participants, and the result we obtained was $5.9\degree$. To further improve the performance, we adopted the 15-second calibration data before each session to fine-tune the model, which resulted in an improved performance of $4.9\degree$. It is worth mentioning that a similar calibration process is also required for commodity eye trackers (e.g., Tobii Pro). The distance between the participants’ eyes and the screen center is measured to be around 60 cm so we have a field of view of $60\degree$ (the largest possible angular error) in this study. We made a demo video showing how our prediction looks like visually with this level of accuracy.

Next, we aimed to compare the impact on gaze tracking performance of using different ground truth acquisition methods: a commodity eye tracker (Tobii Pro Fusion) versus our method (using instruction points on the screen). We used the eye tracking data recorded by Tobii Pro Fusion as the ground truth to train the model, and the MGAE after fine-tuning was $4.9\degree$. We conducted a repeated measures t-test between the results using Tobii data as the ground truth and those using instruction points as the ground truth for all 12 participants, and did not find a statistically significant difference ($p=0.92>0.05$). This suggests that using instruction points on a screen monitor as the ground truth can be as effective as using Tobii data.

Apart from that, we also recorded the eye tracking accuracy of Tobii Pro Fusion itself which was reported after the calibration process of the Tobii platform. The results showed that Tobii Pro Fusion can track the gaze points with an average accuracy of $1.9\degree$ during the calibration process for all participants. We plotted the tracking performance of both GazeTrak and Tobii for all participants in Fig. 4.

In this subsection, we evaluated the impact of the number and placement of microphones on tracking performance to determine the optimal sensor position for the best results. We assessed four different settings: 1) one microphone on each side (left and right); 2) two microphones on each side; 3) three microphones on each side and 4) all four microphones on each side. In the first setting, we compared the performance using data from four sets of microphone settings (M1+M5, M2+M6, M3+M7, and M4+M8 in Fig. 3 (f)), which is presented in Tab. 2. The findings demonstrate that the M4+M8 pair of microphones provides the best tracking performance among the four pairs tested. We conducted a one-way repeated measures ANOVA test on the results of the four settings and identified a statistically significant difference ($F(3,44)=6.74,p=0.001<0.05$). These results indicate that microphone placement can affect gaze tracking performance, possibly due to differences in signal reflection before arriving at different microphones.

We further conducted experiments to evaluate performance using different combinations of microphones under settings 2 and 3. The results showed that the best performance was 5.9 degrees and 5.5 degrees, respectively. We also ran a one-way repeated measures ANOVA test among the results of these four settings using data from 12 participants. The results showed a statistically significant difference ($F(3,44)=51.61,p=0.00001<0.05$). These findings suggest that our system requires four microphones on each side (eight microphones in total) to achieve the best performance.

Blinking can introduce noise in our highly-sensitive acoustic sensing system as it can lead to relatively large movements around the eye. We conducted an evaluation to determine whether blinking affects the tracking performance of our system. For this evaluation, we selected data from three participants with the best, worst, and average tracking performance (P1, P2, P10). We removed the data where the participant blinks (about 10% of total data) based on the ground truth data obtained from Tobii Eye Tracker. We then used the processed data to retrain the user-dependent model for each participant. Our results showed that the performance did not improve after removing the blinking data. One possible reason for this result is that the blinking patterns are consistent and can be learned by the machine learning model. Therefore, our findings suggest that blinking does not significantly impact the performance of our system.

To reduce the need of providing lots of training data for a new user, we employed a three-step process to train a user-adaptive model. Firstly, we trained a large base model using data from all participants except the one being tested. Secondly, we fine-tuned the model using the training data collected from the current participant. Notably, the user only needs to provide training data once during the initial system use. Finally, at the beginning of each session, we further fine-tuned the model using calibration data collected from the participant before testing or using the system. To determine the amount of data required to achieve competitive tracking performance, we reserved two sessions of data for testing and used varying amounts of training data from the participant to fine-tune the large model.

The results show that a new user only needs to provide six sessions of training data (approximately 20 minutes) to achieve good performance. Collecting more data does not necessarily result in better performance. Additionally, with only two or three sessions of data (approximately 6 minutes), the system can achieve a performance of $6.7\degree$ and $6.1\degree$, respectively. If no user data is collected, the performance is $11.3\degree$. This is likely because different people have unique head, face, and eye shapes. Therefore, to further reduce the amount of training data required from each new user, we may need to collect a significantly larger amount of training data from a more diverse set of participants.

To ensure that our acoustic sensing system is resistant to different types of environmental noise, we conducted two experiments as described in this subsection.

In our user study, we only tested our system on one glass frame (F1). However, we believe our GazeTrak system can be easily applied to glasses with different frame styles. In order to validate this assumption, we deployed our system on two other pairs of glasses as shown in Fig. 5. The original glass frame in Fig. 5 (a) is frameless and relatively lightweight. In this study, we applied our system on two new glasses with different styles, size and weight. The first new glass frame (small glasses, F2) has a smaller size than F1 but a larger weight due to the frame around the lens (Fig. 5 (b)). The second new glass frame (large glasses, F3) with a frame around lens (Fig. 5 (c)) has a much larger size and weight compared to F1 and F2. To evaluate our system on these new glass frames, three participants from the original study (P1, P5 and P7) agreed to participate in this additional study. The study setups and procedures were exactly the same as the previous study described in Sec. 5.

We collected 12 sessions of data for each participant testing each glass frame. Since Subsec. 6.4 indicates that 6 sessions of training data is sufficient, we discarded the first 4 sessions for each glass frame and used the last 8 sessions to run a 4-fold cross validation in order to test the tracking performance of our system on different glasses. In this case, we can make sure that participants are familiar with the wearing of all the glass frames and eliminate the impact of some random factors. The evaluation result shows that the small glasses (F2) yielded a similar average performance to the original glasses (F1) (both at $5.3\degree$), while the large glasses (F3) resulted in a relatively poorer average performance (at $6.1\degree$), with a drop in performance of 15%. One possible reason for the performance difference is that the sensors on the larger glasses were much closer to the skin. Sometimes, the sensors may directly touch the skin, which could block the transmission and reception of signals, as we explained in Subsec. 4.2.

In the previous user studies, we evaluated GazeTrak with various configurations under different scenarios, using the initial prototype that we had developed. The results of these studies helped us confirm the prototype settings and develop an optimized system prototype, which features a more compact form factor as shown in Fig. 3 (g). In this subsection, our objective was to assess the performance and power consumption of this final prototype.

To achieve this goal, the deep learning models were trained and synthesized in advance, using the ai8x libraries (AI, 2022). We implemented two models with ai8x, which were ResNet-18 (used in the previous study) and MobileNet for comparison. Due to the hardware limit of MAX78002, we modified the models to be compatible with the chip. Specifically, for a Conv2d layer, the kernel size could only be set to 1x1 or 3x3 and the stride is fixed to [1, 1]. In addition, some convolution layers of ResNet-18 were substituted with depthwise separable convolution layers to avoid exceeding the limit of the number of parameters in the model. Furthermore, we quantized the input and the weights of the models with ai8x, which converted them all into 8-bit data format to save memory for storage and increase the speed of inference.

Before the deep learning model, we also need to apply a band-pass filter on the received signals and perform cross-correlation between received signals and transmitted signals to obtain echo profiles as described in Subsubsec. 3.1.2. In the implementation on MAX78002, to reduce processing time, we removed the band-pass filter since all the computations are done on the MCU and transmitting private data is no longer a concern.

Then we experimented two different methods to realize the cross-correlation: (1) brute force to calculate echo profiles point by point; (2) the dot product function in the CMSIS-DSP library. Results of standard tests (AI, 2023) revealed that it took the system 178.3 ms and 45.4 ms to compute one echo frame and make one inference with these two methods utilized respectively. Considering that one frame of our audio data comes every 12 ms in our system (600 samples $\div$ 50000 samples/s), this processing time is too long to keep our system running in real-time with an FPS of 83.3 Hz. Finally, we explored method (3) a Conv2d layer (kernel size 1x1) with transmitted signals as the untrained weights and received signals placed along the channel axis of the input. This can increase the speed of echo profile calculation because it uses the CNN accelerator on MAX78002. We compressed the samples used for cross-correlation from 600x600 to 34x34 and the pixels of interest from 60 pixels (20.4 cm) to 30 pixels (10.2 cm) in this case to further decrease the processing time.

With this Conv2d layer added on top of the deep learning model, the model directly takes the raw audio data as input in instances with the size of 64 (34+30 samples) x 26 (frames) x 8 (microphones). This method allows the system to make one inference within 10.3 ms, which is enough for the real-time pipeline with a double-buffer method applied (DMA moves the current frame in one buffer while the CPU processes the previous frame in another buffer).

To validate these modifications and compression, we evaluated the in-session performance of different models with different settings using data collected with the final prototype in Subsec. 6.7 and showed the results in Tab. 5.

As shown in the table, the same model trained with ai8x is slightly worse than that trained with PyTorch given the constraints of the convolution layers discussed above. Compressing the size of input data does not affect the accuracy. While MobileNet yields comparable accuracy to ResNet-18, both models suffer a slight performance drop after quantization since the precision of data is decreased.

Given the limitation of the I2S interfaces on MAX78002, to test our system in a more realistic condition, we still use Teensy 4.1 to control the speakers and microphones and transfer the received audio data to MAX78002 via the serial port. To accelerate the transmission speed, only the samples that are used for processing on MAX78002 are transferred. This generates a steady stream of audio data to MAX78002. In future, we will explore connecting microphones directly to MAX78002 using multi-channel audio protocols such as Time-division Multiplexing (TDM). Evaluation results showed that for ResNet-18 and MobileNet, MAX78002 spent 124.1 ms and 41.6 ms respectively loading the weights of the model. This is a one-time effort and can be done before running the real-time pipeline so it did not impact the refresh rate. Then it took 12 ms to load one instance and make an inference based on it in real-time for both ResNet-18 and MobileNet, giving a refresh rate of 83.3 Hz.

We measured the power consumption of the MAX78002 evaluation kit while it made inferences. Tab. 6 demonstrates that MAX78002 consumes 96.9 mW and 86.0 mW respectively when making inferences with ResNet-18 and MobileNet at 83.3 Hz. The refresh rate can be reduced to 30 Hz to save power, which is enough for many applications. In this case, the power becomes 79.0 mW and 75.7 mW respectively.

If we can use MAX78002 to directly control speakers and microphones in future, we will be able to optimize the power efficiency and keep the overall power consumption of our real-time system around 95.4 mW, i.e., 79.0 mW (MAX78002 with ResNet-18 running at 30 HZ) + 16.4 mW (2 speakers and 8 microphones). One should keep in mind that this is just an estimate of the power of this real-time system and the power consumption of MAX78002 might increase if it does need to control the sensors but we do not expect it to be very high because the current power of MAX78002 already includes that of the CPU and the CNN accelerator running at full speed.

We adopted two traditional regression models, which are linear regression (LR) and gradient boosted regression trees (GBRT), to predict gaze positions using the data collected in Subsec. 6.7 and the results showed that the average in-session tracking accuracy for these two models across 10 participants are 11.6$\degree$ and 6.8$\degree$ respectively. Compared to the results in Tab. 3, the traditional regression models output much worse accuracies than ResNet-18 (3.6$\degree$). We conducted an analysis of the impurity-based feature importance with GBRT, comparing the features in different channels of microphones in Tab. 7. It turns out that the channels receiving signals from 18-21 kHz (S1) are generally more important than channels receiving signals from 21.5-24.5 kHz (S2). Furthermore, the microphones that are closer to the inner corners of the eyes (M1, M4, M5 M8) are more important than those closer to the tails of the eyes (M2, M3, M6, M7).

The goal of this paper is to demonstrate the feasibility of our new acoustic-based gaze tracking system on glasses. While our eye tracking accuracy of $4.9\degree$ is comparable to some webcam-based methods, it is lower than commercial eye trackers ($1.9\degree$ in our study). Therefore, our system may not be immediately applicable to some applications requiring highly precise eye tracking. However, our system can still be used in many applications, such as interaction with interface elements like buttons in AR, that do not require very high accuracy eye trackers.

Our system can also be potentially used in tracking irregular eye movements, enabling healthcare applications for monitoring users’ health conditions in everyday life. This requires monitoring the gaze movements throughout the day for analysis in everyday life, instead of just tracking their accurate gaze positions for a few hours in a controlled settings. The low-power and lightweight features of our GazeTrak system make it a good candidate solution to enabling a variety of applications that camera-based eye trackers cannot realize, by continuously understanding user gaze movements in the wild for extended periods. Furthermore, our system can alleviate the privacy concern from users as compared to camera-based methods.