Enhancing brain-machine interface (BMI) control of a hand exoskeleton using electrooculography (EOG)
© Witkowski et al.; licensee BioMed Central Ltd. 2014
Received: 22 July 2014
Accepted: 5 December 2014
Published: 16 December 2014
Brain-machine interfaces (BMIs) allow direct translation of electric, magnetic or metabolic brain signals into control commands of external devices such as robots, prostheses or exoskeletons. However, non-stationarity of brain signals and susceptibility to biological or environmental artifacts impede reliable control and safety of BMIs, particularly in daily life environments. Here we introduce and tested a novel hybrid brain-neural computer interaction (BNCI) system fusing electroencephalography (EEG) and electrooculography (EOG) to enhance reliability and safety of continuous hand exoskeleton-driven grasping motions.
12 healthy volunteers (8 male, mean age 28.1 ± 3.63y) used EEG (condition #1) and hybrid EEG/EOG (condition #2) signals to control a hand exoskeleton. Motor imagery-related brain activity was translated into exoskeleton-driven hand closing motions. Unintended motions could be interrupted by eye movement-related EOG signals. In order to evaluate BNCI control and safety, participants were instructed to follow a visual cue indicating either to move or not to move the hand exoskeleton in a random order. Movements exceeding 25% of a full grasping motion when the device was not supposed to be moved were defined as safety violation. While participants reached comparable control under both conditions, safety was frequently violated under condition #1 (EEG), but not under condition #2 (EEG/EOG).
EEG/EOG biosignal fusion can substantially enhance safety of assistive BNCI systems improving their applicability in daily life environments.
Currently, none of the available non-invasive BMI systems provide sufficient reliability for safe continuous control of motor neuro-prostheses in daily life environments.
Here we tested whether and to what extent integration of EOG can enhance such control and improve safety in controlling a hand exoskeleton for grasping motions.
12 BNCI-naïve healthy volunteers (8 male, 4 female, mean age: 28.1 ± 3.63 years) were invited to the Institute of Medical Psychology and Behavioral Neurobiology at the University of Tübingen, Germany, to participate in a 1-hour experimental session. All participants were right handed as evaluated by the Edinburgh Handedness Inventory , and gave written informed consent before the session. The study protocol was approved by the University of Tübingen’s local ethics committee.
All participants were comfortably seated at a desk while EEG was recorded from 5 conventional EEG recording sites (F3, T3, C3, P3, and CZ according to the international 10/20 system) using an active electrode EEG system (Acti-cap® and BrainAmp®, BrainProducts, Gilching, Germany) with a reference electrode placed at FCz and ground electrode at AFz. EEG was recorded at a sampling rate of 200Hz, bandpass filtered at 0.4-70Hz and pre-processed using a small Laplacian filter. EOG was recorded in accordance to the standard EOG placements at the left and right outer canthus (LOC/ROC) (Figure 2).
In the second part of the calibration, participants were instructed to move their eyes from a fixation cross in front of them to the left or to the right without turning the head upon a visual cue (arrow to the left, arrow to the right). A detection threshold for full left and right eye movements was set at two standard deviations (SD) below the average EOG signal recorded during presentation of the visual cue.
For BNCI control, participants were instructed to look at their hand affixed to a motorized hand exoskeleton and to use visuo-kinesthetic motor imagery of right-hand grasping motions to initialize exoskeleton-driven movements .
The reliability and safety of the BNCI system was evaluated during two conditions: during condition #1, participants used only EEG, while during condition #2, participants used EEG and EOG signals. During both conditions, a visual signal randomly indicated the user to either close (green square) or not to move the device (red square): the two indications were given 24 times each in total separated by inter-trial intervals (ITIs) of 4–6 seconds throughout a time window of approximately 6 minutes (Figure 3). Each indication was displayed for 5 seconds after which the device was driven back to open position. Re-setting the exoskeleton into open position required one second. During BNCI control, EEG signals were processed in 100 ms time windows (sample blocks). Each trial consisted of 50 sample blocks in which the BNCI software evaluated if the SMR-ERD value underwent a given threshold (ERD detection threshold) that was set at two standard deviations above average SMR variance at rest. A complete closing of the exoskeleton was only achieved if movement intention was detected during 43 sample blocks per trial leaving the participant 700 ms after presentation of a green square to initiate SMR-ERD that exceeded the ERD detection threshold. The time delay between the brain signal and the actual movement of the hand exoskeleton due to signal transmission and processing was approximately 125 ms. The speed of the exoskeleton was set to allow a full closing motion within 5 seconds.
During condition #2, exoskeleton motions could be interrupted and reset to neutral (full open) position when EOG signals exceeded the EOG detection threshold. During the trials, the participants were free to use either full left or right eye movements to reset the exoskeleton. After the session, all participants were asked to report any difficulties or discomforts during the control of the device.
To evaluate BNCI control and safety across participants and conditions, motions in % relative to a full closing during presentation of either the green or red square were calculated for each condition. A grasping motion was defined as successful if the hand exoskeleton closed the hand more than 50% while the green square was displayed. This value was chosen under the assumption that the average dimension of the most representative objects grasped in daily life range at approximately 50% of the user’s full hand span. Control of the device was defined as successful if average hand-closing motions during green square presentations exceeded more than 60% of full closing motions (i.e. a movement time > 2.6 seconds per trial) reflecting that a highly significant statistical difference (p < 0.01) of SMR-ERD between task and rest states was reached within this trial. A violation of the safety criterion was defined as a closing motion that exceeded 25% of a full hand closing while a red square was displayed. This value was chosen under the assumption that most daily life objects that are grasped are smaller than 75% of a full hand span so that hand exoskeleton-driven motion could be interrupted within one second before any force is being applied to the object. Participants were not aware of these definitions and received no feedback of performance related to these values.
To improve comparability and as indicator of the EOG features’ impact on BNCI control and safety, also false positive classifications during ITI’s were calculated for both conditions. Statistical significance was assumed when p < 0.05.
True and false positive classification rates across conditions
True positive rate in % (green square)
63.59 ± 10.81
60.77 ± 9.42
False positive rate in % (ITI)
28.72 ± 11.02
22.79 ± 9.43
False positive rate in % (red square)
36.11 ± 10.85
12.31 ± 5.39
Under condition #1 (EEG only), the safety criterion was frequently violated across all participants (in 45.91 ± 26.8% of red square presentations) and some exoskeleton motions reached up to 62% of a full closing motion. Under condition #2 (hybrid EEG/EOG control), safety violations occurred in only 10.14 ± 0.3% of red square presentations and the maximum closing motion during a red square presentation stopped at 28%.
Feedback from the participants
None of the participants reported any discomfort or difficulties during BNCI control. After the experiment, all participants stated that control under condition #1 was more difficult compared to condition #2, and 4 participants reported that unintended motions during condition #1 resulted in some dissatisfaction and frustration over time.
This is the first study that investigated whether fusion of EEG and EOG signals can enhance reliability and safety of continuous brain control of a hand exoskeleton performing grasping motions. While all participants successfully reached control under both conditions, inclusion of EOG control under condition #2 significantly reduced unintended hand exoskeleton motions. While under condition #1 unintended motions frequently exceeded 25% of a full closing motion, the violations of the safety criterion were substantially decreased when EOG control was included under condition #2. This suggests that fusing different biosignals represents a powerful strategy to increase usability and safety of non-invasive assistive motor neuro-prosthesis, for example to control an exoskeleton performing grasping motions in daily life environments. The introduced strategy’s daily life applicability could be even further improved by using a switch mechanism based, for instance, on a sequence of EOG signals (e.g. full eye movements to the left followed by eye movements to the right) turning BNCI control into sleep mode when necessary.
While the main rational for using motor imagery in the described paradigm was to provide intuitive control over a hand exoskeleton in which an imagined or attempted movement of a paralyzed hand becomes translated into a matching motion of an assistive device, use of other brain signals, e.g. SSVEPs or P300  may provide better classification performance. Control of these brain signals, though, is less intuitive and depends on external stimulation that might distract from the object manipulated by the exoskeleton.
In the present study, participants had no previous experience with the use of EOG signals for BNCI control and familiarized with EOG control only during the calibration procedure at the beginning of the session. While eye movements are often left intact in patient populations with severe motor disabilities, e.g. stroke or spinal cord injuries (SCI), validity of these results and their dependence on various factors, e.g. cognitive capacity, attention span or alertness should be investigated in future studies. Also, it is conceivable that hybrid EEG/EOG BNCI control can be improved beyond the level demonstrated in this study if effective training protocols [14, 16], or advanced decoding algorithms, e.g. based on Riemannian geometry , discriminative models  or machine learning are applied.
This work was supported in part by the European Union within the WAY Project FP7/2007-2013 under Grant Agreement 288551, the German Federal Ministry of Education and Research (BMBF # 01GQ0831, # 16SV5840) and the Deutsche Forschungsgemeinschaft (DFG, SO-932-2 & Open Access Publication Fund at the University of Tübingen).
- Millán JDR, Rupp R, Müller-Putz GR, Murray-Smith R, Giugliemma C, Tangermann M, Vidaurre C, Cincotti F, Kübler A, Leeb R, Neuper C, Müller KR, Mattia D: Combining brain-computer interfaces and assistive technologies: state-of-the-Art and challenges. Front Neurosci 2010, 4: 161.PubMedPubMed CentralGoogle Scholar
- Cempini M, Cortese M, Vitiello N: A powered finger-thumb wearable hand exoskeleton with self-aligning joint axes. IEEE/ASME Transact Mechatron 2014, 20: 705-716. doi: 10:1109/TMECH.2014.2315528View ArticleGoogle Scholar
- de Almeida Ribeiro PR, Brasil FL, Witkowski M, Shiman F, Cipriani C, Vitiello N, Carrozza MC, Soekadar SR: Controlling assistive machines in paralysis using brain waves and other biosignals. Adv Hum-Comput Interact 2013,2013(2):1-9.View ArticleGoogle Scholar
- Lu CW, Patil PG, Chestek CA: Current challenges to the clinical translation of brain machine interface technology. Int Rev Neurobiol 2011, 107: 137-160.View ArticleGoogle Scholar
- Blankertz B, Dornhege G, Schäfer C, Krepki R, Kohlmorgen J, Müller KR, Kunzmann V, Losch F, Curio G: Boosting bit rates and error detection for the classification of fast-paced motor commands based on single-trial EEG analysis. IEEE Trans Neural Syst Rehabil Eng 2003, 11: 127-131. 10.1109/TNSRE.2003.814456View ArticlePubMedGoogle Scholar
- McFarland DJ, Anderson CW, Müller KR, Schlögl A, Krusienski DJ: BCI Meeting 2005--workshop on BCI signal processing: feature extraction and translation. IEEE Trans Neural Syst Rehabil Eng 2006,14(2):135-138. 10.1109/TNSRE.2006.875637View ArticlePubMedGoogle Scholar
- Pfurtscheller G, Solis-Escalante T, Ortner R, Linortner P, Müller-Putz GR: Self-paced operation of an SSVEP-Based orthosis with and without an imagery-based “brain switch:” a feasibility study towards a hybrid BCI. IEEE Trans Neural Syst Rehabil Eng 2010, 18: 409-414.View ArticlePubMedGoogle Scholar
- Yong X, Fatourechi M, Ward RK, Birch GE: The design of a point-and-click system by integrating a self-paced brain–computer interface with an Eye-tracker. IEEE J Emerg Sel Top Circ Syst 2011, 4: 590-602.View ArticleGoogle Scholar
- Punsawad Y, Wongsawat Y, Parnichkun M: Hybrid EEG-EOG brain-computer interface system for practical machine control. Conf Proc IEEE Eng Med Biol Soc 2010, 2010: 1360-1363.PubMedGoogle Scholar
- Wang H, Li Y, Long J, Yu T, Gu Z: An asynchronous wheelchair control by hybrid EEG–EOG brain–computer interface. Cognit Neurodynamics 2014,8(5):399-409. 10.1007/s11571-014-9296-yView ArticleGoogle Scholar
- Oldfield RC: The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia 1971,9(1):97-113. 10.1016/0028-3932(71)90067-4View ArticlePubMedGoogle Scholar
- Schalk G: Effective brain-computer interfacing using BCI2000. Conf Proc IEEE Eng Med Biol Soc 2009, 2009: 5498-5501.PubMedGoogle Scholar
- Pfurtscheller G, Aranibar A: Evaluation of event-related desynchronization (ERD) preceding and following voluntary self-paced movement. Electroencephalogr Clin Neurophysiol 1979,46(2):138-146. 10.1016/0013-4694(79)90063-4View ArticlePubMedGoogle Scholar
- Soekadar SR, Witkowski M, Mellinger J, Ramos A, Birbaumer N, Cohen LG: ERD-based online brain-machine interfaces (BMI) in the context of neurorehabilitation: optimizing BMI learning and performance. IEEE Trans Neural Syst Rehabil Eng 2011,19(5):542-549.View ArticlePubMedGoogle Scholar
- Sakurada T, Kawase T, Takano K, Komatsu T, Kansaku K: A BMI-based occupational therapy assist suit: asynchronous control by SSVEP. Front Neurosci 2013, 7: 172.View ArticlePubMedPubMed CentralGoogle Scholar
- Soekadar SR, Witkowski M, Birbaumer N, Cohen LG: Enhancing Hebbian learning to control brain oscillatory activity. Cereb Cortex 2014. (in press), doi:10.1093/cercor/bhu043Google Scholar
- Barachant A, Bonnet S, Congedo M, Jutten C: Multiclass brain-computer interface classification by Riemannian geometry. IEEE Trans Biomed Eng 2012,59(4):920-928.View ArticlePubMedGoogle Scholar
- Delgado Saa JF, Çetin M: Discriminative methods for classification of asynchronous imaginary motor tasks from EEG data. IEEE Trans Neural Syst Rehabil Eng 2013,21(5):716-724.View ArticlePubMedGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.