Car Racing Driver Distraction Detection Using Brain Eeg

Transcript

Car Racing Driver Distraction Detection Using Brain EEG Mahsa Salehi Geoffrey Mackellar Christopher Leckie Department of Computing and Information Systems The University of Melbourne Victoria 3010, Australia Emotiv Research Inc Sydney, Australia Department of Computing and Information Systems The University of Melbourne Victoria 3010, Australia [email protected] [email protected] ABSTRACT Monitoring driver attention has a direct effect on decreasing injury/fatality rates. Hence, detecting episodes of driver distraction is fundamental in reducing car crashes. This is particularly significant in car racing environments where reaction times are short, and distraction leads to a reduction in driver’s performance during a race. Considering the limitations of video/speech processing approaches in car racing conditions (wearing fire protective suits and helmets, noisy environments), in this paper we focus on the feasibility of Brain Computer Interface (BCI) approaches, by using a commercial headset which generates brain EEG signals and gyroscope data wirelessly. Our experimental results show the combination of EEG and gyroscope data can lead to a highly accurate distraction detection framework. In addition, our proposed framework is scalable and can potentially be parallelized in order to reduce response times and monitor racing drivers in real-time. Keywords Car Racing, Brain Computer Interface, Electroencephalography (EEG) Signal, Driver Distraction, Data Stream Classification, Real-time Response 1. INTRODUCTION Based on research by the Alberta Transportation annual report in 2011 [1], “distracted drivers are 3 times more likely to be in a crash than attentive drivers” [2]. This suggests that a considerable number of road fatalities are attributable to driver inattention. In terms of car-racing, it is even more important to detect driver distraction episodes. First, if drivers are distracted their performance is likely to decrease. Second, driver distraction might lead to catastrophic crashes at high speeds. Furthermore, tracking distraction information for a driver can be used to further train the driver and improve his/her driving skills. While researchers have mostly considered the optimal performance of car operation in racing, monitoring the driver’s Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Copyright 20XX ACM X-XXXXX-XX-X/XX/XX ...$15.00. [email protected] state during car racing has received little attention. In order to detect a driver’s mental state, such as distraction, drowsiness or emotional states, a variety of video processing (e.g., eye tracking) and speech processing approaches have been proposed. However, in car racing environments drivers should wear fire protective suits and helmets which cover their face and head, hence passive video processing approaches are not practical in such situations. Due to the high level of background noise in the environment, speech processing techniques become ineffective as well. As a result, in this paper we investigate the feasibility of a BCI approach to detect distraction episodes of drivers in car racing environments. We propose analyzing the brain EEG signals of a driver to infer episodes of distraction. We also investigate how the movement data of a driver’s head can further help to detect distraction in drivers. We use a commercial headset which can be put into a driver’s helmet to collect EEG signals from the driver’s brain and gyroscope data from the driver’s head wirelessly. We categorize the distraction scenarios in car racing into three main classes and propose scalable data mining algorithms to detect them. To the best of our knowledge, none of the previous approaches detect episodes of car racing driver distraction using electrical brain activity and gyroscope data. 2. RELATED WORK In order to detect driver distraction, researchers have traditionally focused on video processing techniques [14] and tracking eye movement [15]. However, these approaches can not be used in car racing where drivers faces are covered by a helmet or mask, unless the eye cameras are integrated into the equipment, which has not been studied to our knowledge. More recently, researchers have focused on monitoring the EEG signals of a driver’s brain to find patterns of distraction. In this sense, Lin et al. [9] used a 32-channel EEG system from Neuroscan Compumedics Ltd. to investigate driver cognition. Later, Wang et al. [13] used the same system while considering two types of distractions in their experiments, and achieved high detection accuracy using different machine learning techniques. In [8], 30 electrodes are used to study the dual task of driver performance, and they found an increase in the theta band power of EEG signals in the frontal areas of the brain during distraction. In addition, the authors in [10] also show an increase in the theta band by studying the effects of iPod usage and text messaging on driver distraction. In [11], the effect of alpha spindle in EEG on attention was studied using 32 electrodes from ActiCap, Brain Products GmbH. Finally in a recent study, the authors in [12] found patterns of specific driver distractions, i.e., the beginning and ending of map reviewing, using a Neuroscan system including 36 EEG channels. None of the above EEG-based approaches have considered driver distraction detection in car racing environments. However, Katsis et al. [7] have recently proposed a system to detect the emotional state of drivers in car racing. They use 16 facial EMGs, EDR, ECG and driver respiration data to detect stress level of drivers in car racing with 76.3% accuracy, whereas in this paper our focus is on analyzing EEG and head motion signals to detect distraction episodes of drivers in car racing. 3. OUR METHODOLOGY We have used the Emotiv EPOC headset1 to record EEG signals. This wireless lightweight headset can be adapted to fit into the driver’s helmet and provide monitoring of 14 channel positions of a driver’s brain in real-time. The EPOC headset and its channel positions on the scalp are shown in Fig. 1a-b. In addition, the headset has a 2-D gyroscope to measure head movement. In our framework, the driver distraction scenarios in car racing are categorized into three main classes: 1) When a driver is communicating to someone in the car (e.g., the navigator), 2) When a driver is communicating to someone remotely (e.g., the support team via radio) and 3) Problem solving activities (e.g., planning fuel consumption, strategy and tactics). In this section, we explain our methodology to extract features from EEG signals. Thereafter we propose two approaches to detect the episodes of distraction. First, a shared mixture classifier is proposed. Second, a more scalable algorithm is proposed by using an ensemble based approach. In order to show the robustness of our proposed methods, we also use a boosting algorithm and detect the distraction of drivers. Feature Extraction The signals are generated at 128Hz and data was band-pass filtered at 4-40Hz to reduce ocular artifacts, epoched into trailing 2 second windows (updated at 0.5s intervals) and then FFT was applied. Features were selected based on the usual frequency bands (theta, alpha, low beta, high beta, gamma) and the average power per band was recorded for each sensor. In addition the peak power and peak frequency in each band were recorded at each sensor. Additional features were generated based on accumulated power in regions such as left and right frontal regions, left and right hemispheres, left and right temporal/parietal and occipital power within each specified band, and by accumulating EEP power over the main processing bands (beta and gamma). This results in 266 different features. Additionally, there is another feature extracted from the gyroscope, indicating the number of times the user looked around in a 15 seconds window. Shared Mixture Classifier In order to select the most effective features from the 267 features, we first use canonical variate analysis [4], which produces vectors in the direction of greatest variance between each class and the remaining data. We select the features by sorting the features using the magnitude of each projection coefficient. Thereafter, we use a shared mixture classifier (SMC) [6], allowing all 1 http://www.emotiv.com/epoc.php (a) (b) Figure 1: EPOC (a) Headset, (b) 14-Channels of the mixture components to contribute to the model. In this model each mixture has its own set of likelihoods for belonging to each output class. Here we used 5 mixture components, selected on the basis of residual performance improvement to avoid over training. Random Forest In high dimensional, high volume data sets, random forests (RFs) [3] can be used for both feature selection and classification. In addition, since the trees in the RF model are independent from each other, each tree can be put on different nodes. As a result of such modeling, distraction detection can be parallelized. Hence, this approach is highly scalable in terms of time complexity and is well suited to our real-time application. Therefore, in this paper the RF algorithm is used to first select the most important features and then to detect distraction episodes. AdaBoost Besides using a bagging approach, we used another ensemble technique, AdaBoost [5], to ensure the robustness of our model. In AdaBoost, like other ensemble techniques, a number of classifiers are built. In each round of building the classifier, the misclassified observations gain higher weights for the next round to have a higher probability to participate in building the next classifier. We build a decision tree using a random subset of features in each round of the algorithm in this paper. This approach is also scalable and is well suited to our problem. 4. EVALUATION In this section, we apply the methodology that is described in the previous section in order to classify data streams into attention/distraction. We first describe our data set and then the performance measures we used to evaluate our methodology. All the implementations are in Matlab R2014b and all experiments were done on a Core i7-2600 CPU 3.40GHz running Windows 7. We have used the implementation of Breiman’s random forest algorithm [3] and AdaBoost [5] in Matlab R2014b. Data Set There were 18 subjects who attended a driving simulation which took about 40 minutes each. We asked the subjects to blink a few times before starting the experiment. The rationale with the repeated blinks is to make sure there are reference marks aligning the eye tracker, video and EEG channels to make sure we align the events correctly. For the purposes of experiments because of the difficulty of obtaining actual racing conditions, we have provided more controlled conditions to normal drivers instead. To do so, for each of the three distraction categories mentioned earlier, we have used the following controlled tasks: 1. Communicating to navigator: Talking to passenger 2 1.5 1 0.5 3 AUC OOB Accuracy 90 2.5 2 1.5 1 0.5 80 70 0 0 50 100 150 200 250 0 Feature Index (a) EEG features 50 100 150 200 250 60 Feature Index 266 (b) EEG+Gyro features Figure 2: Feature importance using random forests and OOB error AUC(%) 73.5 73.5 73 73 72.5 72 4.1 Results and Discussion The experiments are divided into two main categories. In the first category we consider just the EEG related features and exclude the gyroscope feature. In this category, the aim is to find out to what extent the brain signals can help us to find the subject’s distraction. In the second category, we add the gyroscope feature in order to improve the detection accuracy. In the random forest algorithm, initially, the forest size is set to 100, the size of the subset of features is set to 50, and the minimum number of data points per tree leaf is set to 1. 4.1.1 Feature Selection - RF We used the training data set and built random forests for both EEG and EEG+Gyro feature sets. Thereafter, the importance of each feature is calculated based on the increase 72.5 72 71.5 71.5 100 The resulting data set contains 180,000 records with 267 features. We also recorded the video of the experiments to be used for labeling the distraction episodes in the data set by a human expert. A hold out evaluation methodology is used 2 and of subjects are chosen randomly for training purposes 3 and the rest of the subjects are left for evaluation. We have rescaled all features in the data set, using zero-mean and unit-variance. Performance Measure The first performance measure is AUC (Area Under Curve) of the ROC curve, which is a curve showing the relationship between false positive rates and true positive rates in our evaluation (testing) set. The output of the classification algorithms is the probability of being distracted, and each data point on the ROC curve is related to the threshold of the output probabilities. Hence, this is the detection accuracy and in this paper we use AUC and detection accuracy interchangeably. Since we have used a random forest algorithm in our methodology, the out of bag (OOB) accuracy is the second measure we use for the evaluation. The OOB accuracy is the classification accuracy of observations that are not used in building a tree. In order to compute OOB accuracy for each such observation, the trees that do not use such observations are chosen. Hence, it is different to the classification accuracy over the whole training data set. 20 Figure 3: Feature reduction using random forests 2. Communicating to the remote support team: Mobile call, Recording call 3. Problem solving activities: Solving simple mental arithmetic challenges placed on the road, assessed by the driving through the chosen answer from a range of alternatives on screen 25 Number of Features AUC(%) 0 3.5 Accuracy(%) Out-of-Bag Feature Importance Out-of-Bag Feature Importance 2.5 200 400 Forest Size (a) 1000 1 5 10 Minimum Number data points in Leaf (b) Figure 4: Effect of (a) forest size on AUC, (b) number of data points per tree leaf on AUC in prediction error if the values of that feature are permuted across the OOB observations. Fig. 2a-b show the feature importance for each category. As depicted, the feature importance in both categories are roughly similar and related to the processing power and peak frequencies in different locations of the brain. In addition, the gyroscope feature in the second category (the last bar in the rightmost of Fig. 2b) is the most important feature in this category. Hence, we expect the detection accuracy would increase by using this feature. Using the feature importance measure can help us to select the most important features in the classification task. We have selected the features with an OOB feature importance measure greater than 1 and 1.5 over the first category of feature sets which results in the top 25 and top 20 features respectively. Fig. 3 depicts the AUC results and OOB accuracy results using all features in category one, the top 25 features and the top 20 features. We notice a slight decrease (0.48%) in the AUC by reducing the number of features to 25. However, if we keep reducing the features to 20, the AUC drops 1.62%. In addition, the OOB accuracy decrease in the 25 feature set is only 0.07%. If we keep reducing to 20 features, the OOB accuracy drops 1.42%. The results suggest that the top 25 features can be considered as a useful discriminator set, and we continue our experiments with this set of features. In addition to keeping the AUC and OOB accuracy roughly the same, selecting the top features reduces the computational complexity and response time, which is significant in this application. Note that we similarly select the features for the second category of feature sets using Fig. 2b. We now consider the new reduced feature set (top 25) to investigate the effect of the parameter settings of the random forest algorithm on AUC in Fig. 4. In Fig. 4a, we increase the number of ensembles (forest size) in the experiments. As we construct more trees in the forest, the AUC of the test set increases. In this figure there is a sudden increase in 74 81 to the effectiveness of deep learning approaches on complex signals, we will apply them to find distraction patterns in EEG signals. 73.5 AUC(%) AUC(%) 80 73 72.5 79 78 6. 77 [1] Alberta Transportation Annual Report. [Online]. Available: http://www.transportation.alberta.ca. [2] http://distracteddriving.caa.ca/education/. [3] L. Breiman. Random forests. Machine Learning, 45(1):5–32, 2001. [4] R. B. Darlington, S. L. Weinberg, and H. J. Walberg. Canonical variate analysis and related techniques. Review of Educational Research, pages 433–454, 1973. [5] Y. Freund and R. E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55(1):119–139, 1997. [6] G. A. Jarrad and D. W. McMichael. Shared mixture distributions and shared mixture classifiers. In Information, Decision and Control, pages 335–340, 1999. [7] C. Katsis, Y. Goletsis, G. Rigas, and D. Fotiadis. A wearable system for the affective monitoring of car racing drivers during simulated conditions. Transportation Research Part C: Emerging Technologies, 19(3):541–551, 2011. [8] C.-T. Lin, S.-A. Chen, T.-T. Chiu, H.-Z. Lin, L.-W. Ko, et al. Spatial and temporal EEG dynamics of dual-task driving performance. Journal of Neuroengineering and Rehabilitation, 8(1):11–23, 2011. [9] C.-T. Lin, L.-W. Ko, and T.-K. Shen. Computational intelligent brain computer interaction and its applications on driving cognition. Computational Intelligence Magazine, 4(4):32–46, 2009. [10] M. Mouloua, A. Ahern, A. Quevedo, D. Jaramillo, E. Rinalducci, J. Smither, P. Alberti, and C. Brill. The effects of iPod and text-messaging use on driver distraction: a bio-behavioral analysis. Work: A Journal of Prevention, Assessment and Rehabilitation, 41:5886–5888, 2012. [11] A. Sonnleitner, M. Simon, W. E. Kincses, A. Buchner, and M. Schrauf. Alpha spindles as neurophysiological correlates indicating attentional shift in a simulated driving task. International Journal of Psychophysiology, 83(1):110–118, 2012. [12] S. Wang, Y. Zhang, C. Wu, F. Darvas, and W. A. Chaovalitwongse. Online Prediction of Driver Distraction Based on Brain Activity Patterns. Transactions on Intelligent Transportation Systems, 16(1):136–150, 2015. [13] Y.-K. Wang, T.-P. Jung, S.-A. Chen, C.-S. Huang, and C.-T. Lin. Tracking attention based on EEG spectrum. In HCI International 2013-Posters Extended Abstracts, pages 450–454. 2013. [14] C. Wege, S. Will, and T. Victor. Eye movement and brake reactions to real world brake-capacity forward collision warnings - A naturalistic driving study. Accident Analysis & Prevention, 58:259–270, 2013. [15] Y. Zhang, E. Harris, M. Rogers, D. Kaber, J. Hummer, W. Rasdorf, and J. Hu. Driver distraction and performance effects of highway logo sign design. Applied Ergonomics, 44(3):472–479, 2013. 72 71.5 RF Ada SMC (a) EEG features RF Ada SMC (b) EEG+Gyro features Figure 5: AUC of different algorithms on both categories of features AUC of 0.34% from 100 to 200 ensembles. However, the increase from 400 to 1000 ensembles only gives us a 0.09% rise in AUC. Fig. 4b shows the effect of the minimum number of data points per tree leaf. We notice that by increasing this parameter the AUC slightly increases by roughly 0.08%. As a result, these two parameter values can be set according to the response time requirements to get more accurate predictions. In addition, in our AdaBoost algorithm, the same feature set and same parameter settings as RF are used. 4.1.2 Feature Selection - SMC As discussed in the previous section, we use canonical transformation based on the strongly labeled data to sort the features using the magnitude of each projection coefficient. In the experiments we select the top 100 features for classification by SMC, which decrease the response time. The summary of AUC results of applying RF, AdaBoost and SMC on both categories of features are shown in Fig. 5ab. In Fig. 5a the AUC measure for all methods are essentially high, which shows the ability of detecting distraction episodes based on electrical brain signals. In addition, RF is the most accurate method with an AUC equal to 73.53%. However, as expected, adding the gyroscope feature increases the detection accuracy in Fig. 5b. While the AUC in all methods is quite high, RF is again the most accurate method with an AUC of 81.1%. In addition, all proposed methods are scalable and can be parallelized, which is an important advantage in real-time scenarios. 5. CONCLUSION AND FUTURE WORK In this paper we have investigated the feasibility of detecting car racing driver distraction episodes in real-time using EEG signals from the driver’s brain. We proposed an accurate and scalable methodology to this problem. We used feature selection approaches which reduce the response time, as well as three classification algorithms. Our experimental results on all three methods suggest that the driver distraction patterns can be detected in real-time using 14 channels of EEG signals with high accuracy. Moreover, random forests are the most accurate method with a detection accuracy of 73.53%. We found that the most important features in our experiments are related to the processing power and peak frequencies in different locations of the brain. In addition, we show that by adding gyroscope data to the EEG signals, the detection accuracy of our framework increased to 81.1%. Using this methodology, racing car drivers can be trained to manage their distractions. Moreover, it can provide a framework to help with better planning of real-time tasks depending on the driver’s distraction state in the race. In the future, we intend to investigate the planning methods in combination with the results of this paper to increase the performance of drivers during races. In addition, due REFERENCES