Manipulation of visual biofeedback during gait with a time delayed adaptive Virtual Mirror Box
© Barton et al.; licensee BioMed Central Ltd. 2014
Received: 18 July 2013
Accepted: 2 June 2014
Published: 10 June 2014
A mirror placed in the mid-sagittal plane of the body has been used to reduce phantom limb pain and improve movement function in medical conditions characterised by asymmetrical movement control. The mirrored illusion of unimpaired limb movement during gait might enhance the effect, but a physical mirror is only capable of showing parallel movement of limbs in real time typically while sitting. We aimed to overcome the limitations of physical mirrors by developing and evaluating a Virtual Mirror Box which delays the mirrored image of limbs during gait to ensure temporal congruency with the impaired physical limb.
An application was developed in the CAREN system’s D-Flow software which mirrors selected limbs recorded by real-time motion capture to the contralateral side. To achieve phase shifted movement of limbs during gait, the mirrored virtual limbs are also delayed by a continuously calculated amount derived from past gait events. In order to accommodate non-normal proportions and offsets of pathological gait, the movements are morphed so that the physical and virtual contact events match on the mirrored side. Our method was tested with a trans-femoral amputee walking on a treadmill using his artificial limb. Joint angles of the elbow and knee were compared between the intact and mirrored side using cross correlation, root mean squared difference and correlation coefficients.
The time delayed adaptive virtual mirror box produced a symmetrical looking gait of the avatar coupled with a reduction of the difference between the intact and virtual knee and elbow angles (10.86° and 5.34° reduced to 4.99° and 2.54° respectively). Dynamic morphing of the delay caused a non-significant change of toe-off events when compared to delaying by 50% of the previous gait cycle, as opposed to the initial contact events which showed a practically negligible but statistically significant increase (p < 0.05).
Adding an adaptive time delay to the Virtual Mirror Box has extended its use to treadmill gait, for the first time. Dynamic morphing resulted in a compromise between mirrored movement of the intact side and gait events of the virtual limbs matched with physical events of the impaired side. Asymmetrical but repeatable gait is expected to provide even more faithful mirroring.
Visual illusions provided by a mirror box can reduce phantom limb pain (PLP) experienced by amputees . These illusions are created by placing a mirror in the mid-sagittal plane of the person so that the amputated limb appears intact. Regular exposure to such illusions is termed ”Mirror Therapy”, and when used repeatedly, it affords continuous pain relief in a substantial percentage of PLP sufferers [2–5]. The benefits of mirror therapy are not exclusive to PLP, it has also been used for motor rehabilitation of stroke [6, 7], complex regional pain syndrome (CRPS)  and cerebral palsy , although the successful use of mirror therapy in conditions other than PLP is mainly evidenced in small sample case studies . Despite this, mirror therapy shows considerable promise to the extent that the presence of a physical mirror may not actually be necessary as similar effects have been shown using motor imagery  and manipulated video footage . These examples strongly point to a significant role for the visual system in chronic pain and in motor rehabilitation, all of which can be exploited with virtual reality (VR).
Mirror therapy has been provided using virtual reality in a number of different ways. In sitting upper limb amputees, reflected digital video projected onto a screen above the missing limb successfully decreased PLP following training twice a week for eight weeks. Reduction of pain was maintained four weeks after the intervention . An immersive VR protocol was developed to provide mirror therapy . This protocol overcame a weakness of traditional mirror therapy in that it did not require a mirror to be kept in the body’s mid-sagittal plane, or to keep the head turned towards the mirror. All of these studies indicate that if the visual stimulus is sufficiently powerful, the use of VR mirroring for chronic pain and motor rehabilitation is worth investigating.
Whilst all of the aforementioned studies have shown considerable potential for pain reduction and motor rehabilitation, they share the weakness that using either a physical mirror or a fully immersive virtual environment restricts the choice of tasks due to the participants’ seated position. A physical mirror allows only spatial mirroring and so this prohibits phase shifted or parallel reaching type movements  which would require a temporal delay. Furthermore, open kinetic chain symmetrical limb movements while sitting with the limbs in the air have limited real-life equivalents and therefore lack ecological validity. Importantly, chronic pain occurs not only in static but also during dynamic activities, in fact exercise may even be used as a coping strategy .
The extension of mirror therapy to activities such as walking gait has been avoided because performing such functional activities is not feasible within the constraints of a physical mirror box or fully immersive VR. The mirroring of limbs during walking would give beneficial real-time visual feedback to patients not only with chronic pain e.g. PLP, CRPS but also those with motor impairment e.g. stroke or cerebral palsy during a functional activity of daily living. Through techniques such as real-time three-dimensional motion capture and partially immersive VR, motor impairment can be assessed and mirror therapy administered to benefit such populations during dynamic activities. The aim of this study was to develop a Virtual Mirror Box application for gait by overcoming the limitations of the conventional mirror box.
In order to mirror one side of the body to the other side, a second type of virtual mirror is necessary. The custom made Virtual Mirror Box (VMB) within D-Flow’s MoCap module  mirrors the arm and/or leg of a selected side in place of the other side using a time delay as input. Feeding zero delay to VMB replicates the functionality of a physical mirror box with the added possibility of positioning the avatar anywhere by moving the display. In order to produce time delayed mirroring, an application created in D-Flow by linking various processing modules calculates initial contact and toe-off events of both sides in order to determine the time delay which is applied to the mirrored arm and/or leg during gait. The simultaneously applied spatial mirroring and temporal delay ensures that movement of the virtual mirrored limb matches movement of the impaired physical limb as closely as possible. Such spatial and temporal congruence between the virtual and physical limb is necessary for the patient to develop a sense of ownership of the virtual limbs.
Technical challenges and solutions
Detection of gait events
The time delay required for mirroring limbs during gait is essentially the phase shift between movement of the right and left sides, which can be calculated from recurring events identified during the cyclic activity of gait. There are numerous methods described in the literature to determine initial contact and toe-off events from the movement of markers attached to the legs (e.g. [17–20]) but it is unlikely that any one of the methods will be applicable to all forms of gait given the diversity of the targeted conditions.
Considering that any recurring event can be used to calculate the delay, we chose the ‘coordination algorithm’  which uses the foremost and rearmost position of the heel marker relative to the sacrum marker (in the direction of progression) as estimates of initial contact (IC) and toe-off (TO) respectively. To detect these gait events, the differences of antero-posterior positions of the left heel to sacrum and right heel to sacrum markers were passed to a 5 Hz low-pass second order Butterworth Filter module. Rising zero crossing of the filtered derivatives (frame-to-frame marker displacements) was detected by monitoring if the actual value was > =0 and the previous value was <0 triggering right and left IC events (RIC and LIC). Detection of falling zero crossing (actual value < =0 and previous value >0) was used to identify right and left TO events (RTO and LTO).
Delay of mirrored side
Participant and protocol
Gait of a left sided trans-femoral amputee (age 49 years; height 1.79 m; mass 76 kg; 6 years since amputation) was recorded using the HBM, captured by a 16 camera Vicon MX system while walking at his preferred speed on a treadmill at 2.8 km · h-1 for 83 s. The amputee used a KX-06 knee and an Echelon ankle-foot device, both controlled hydraulically (Chas A Blatchford & Sons Ltd, Basingstoke, UK). The VMB software module of the CAREN system mirrored and delayed the right leg and arm markers with reference to mirror planes attached to the pelvis and trunk respectively. Flexion/extension angles of the knees and elbows were re-sampled off-line at 100 Hz with quintic splines to correct for the non-constant sampling rate of the D-Flow software. Cross correlation of angles between the right and left sides was used to quantify lag over a 10 s long clean section (35-45 s) using Matlab 2012b (The Mathworks, Natick MA, USA). Mirroring accuracy was quantified by Root Mean Squared Deviations (RMSD) of the lag-corrected, mirrored and delayed angles of the knees and elbows together with correlation of the corresponding angles. To test the effect of smoothly changing the mirroring delay, IC and TO events of the physical left side and the mirrored and delayed right side were compared with paired t-tests over a section between 24-82 s when gait was in a steady state. All procedures were approved by the institutional ethics committee (11/SPS/037) and the participant provided written informed consent.
The ability of the Virtual Mirror Box to replace the impaired side with virtual limbs was evaluated by comparing movement of an amputee’s impaired limbs to the movement of his virtual limbs generated by the VMB. Specifically the VMB was expected to provide a faithful correction of the movement asymmetry achieved by mirroring and delaying the reference side using a continuously changing adaptive time delay to predict gait events of the mirrored side.
Effects on the movement of limbs
Visually the virtual leg showed normal movement characteristics taken from the participant’s unaffected leg including normal knee flexion in stance and normal alignment of the whole leg. Asymmetry of the arm movements was also corrected. Using the VMB more than halved the side-to-side RMSD of knee and elbow angles from 10.86° and 5.34° to 4.99° and 2.54° respectively. Correlation coefficients used as measures of symmetry improved from 0.96 and 0.58 to 0.99. The RMSD values found in this study are comparable but higher than our previous findings with normal gait where the RMSD between the original and virtual knees and elbows was 3.5° and 0.9° respectively . This difference may be explained by the unimpaired participant’s more repeatable gait cycles.
Morphing inevitably makes the angles slightly distorted but this is an inevitable compromise for the benefit of temporal congruency between physical and virtual events. For example the RMSD between the physical right leg and virtual left leg is higher after morphing (4.99°) than with mirroring only (without morphing) (2.65°) but in return, the physical and virtual events are expected to match better.
Effects on gait events of the mirrored side
The application of smooth transitions between mICdelay and mTOdelay handles asymmetrical gait characterised by an offset and altered proportions of stance and swing phases on the contralateral side. This method however results in a perfect match of virtual and physical events on the mirrored side only if mICdelay and mTOdelay remain the same over consecutive gait cycles. Gait and especially abnormal gait however exhibits cycle-to-cycle variation in the timing of gait events and so the application of mICdelay and mTOdelay derived from the previous gait cycles inevitably causes minor discrepancies between the virtual and physical events.
In case of the amputee we tested, the effects of morphing did not reduce the difference between physical and virtual events any better than using 50% of the previous gait cycle as a delay. This can be explained by the cycle-to-cycle variability of gait confirmed by the cycle-to-cycle changes of mICdelay and mTOdelay (c.f. Figure 7).
Morphing of the delay is preferred over using 50% of the previous gait cycle duration. Firstly, the morphing method causes only a non-significant difference between physical and virtual TO events even if there is cycle-to-cycle variability of gait. The difference between physical and virtual IC events is statistically significant but is only 0.016 s on average, which is only 1.1% of the mean left stride time (1.4 s) and therefore not practically meaningful. Secondly, if the cycle-to-cycle variability was lower, then the difference between the physical and virtual events would be expected to reduce as well as the RMSD between the physical and virtual limbs. If cycle-to-cycle variability of the patient’s gait is available then this can determine if dynamic morphing or 50% of the previous gait cycle duration should be used.
The roles of mirror planes
In its current implementation of the VMB algorithm, markers of the leg are mirrored to a vertical plane which follows rotation of the pelvis and is attached to the sacrum marker. The hip joint centre on the mirrored side, however, is derived from the pelvis markers which are not mirrored. As a consequence of the hip centre moving together with the pelvis and not the mirrored thigh, the position and orientation of the thigh segment is not an exact mirrored image of the other side when pelvic movements are asymmetrical. This limitation of the mirroring method explains the minor difference found in the knee angle when mirroring, as our participant had considerable asymmetry in pelvic movements. The difference in knee (and hip) angles could be eliminated by mirroring the hip joint but that would cause an apparent dislocation of the mirrored hip joint, which is perhaps more disadvantageous visually than the minor difference in the position and orientation of the thigh and consequential asymmetry of the knee and hip angles.
Contrary to the leg, the arm is mirrored to a trunk-attached mirror plane defined by markers over C7 and T10 spinous processes and the sternum. As a result the mirrored arm markers move together with the shoulder centre better than the leg markers with the hip centre even when there is much lateral trunk sway. Such a definition of the trunk mirror plane also explains why the elbow angles were almost identical when mirroring the arm. A disadvantageous consequence of the trunk mirror plane being attached to the trunk’s local sagittal plane is that with lateral trunk sway the lateral displacement of the arm is amplified. This visual artefact can be eliminated by optionally defining a trunk mirror plane in a vertical plane (similarly to the mirror plane of the pelvis). Such mirroring would however introduce virtual dislocation of the shoulder, similar to the hip joint. Exact effects of using different mirror planes and their impact on visualisation of the avatar should be evaluated in future studies.
Visual presentation of the virtual avatar
The avatar following movement of the participant manipulated by the VMB algorithm can be presented in a number of ways. A TV screen positioned in front of the treadmill in a portrait orientation replicates the setup commonly used in physiotherapy rooms or fitness centres where mirrors are in front of the treadmills. In addition to the default straight on view, any viewing angle can be shown by rotating, panning and zooming with the virtual camera in D-Flow. A special effect can be achieved by assigning a local coordinate system to a TV display by attaching three reflective markers. The virtual camera in D-Flow can then be linked to the physical display’s position and orientation. A display moved around the participant on wheels can show a changing view as if it was a physical mirror carried around. A head mounted display can also be used to present the manipulated avatar to the participant although the head has to be tilted forwards to see enough of the mirrored leg. Such manipulations of the display can provide numerous options for presenting their movement to participants, some of which may improve the effectiveness of visual feedback .
Thanks to an adaptive time delay calculated continuously from past gait events by the Virtual Mirror Box, for the first time patients with unilateral gait problems can view their own symmetrical gait while walking on a treadmill. A number of inevitable limitations exist due to predicting gait events from variable gait, but overall the VMB can be regarded as a powerful tool for providing visual feedback to patients with a variety of conditions during gait. The clinical effect of such dynamic visual feedback can be evaluated next in amputees, cerebral palsy, stroke and complex regional pain syndrome.
We gratefully acknowledge the funding received from the Institute for Health Research of Liverpool John Moores University in support of developing the Virtual Mirror Box.
- Ramachandran VS, Rogers-Ramachandran D: Synaesthesia in phantom limbs induced with mirrors. Proc Biol Sci 1996,263(1369):377-386. 10.1098/rspb.1996.0058View ArticlePubMedGoogle Scholar
- Brodie EE, Whyte A, Waller B: Increased motor control of a phantom leg in humans results from the visual feedback of a virtual leg. Neurosci Lett 2003,341(2):167-169. 10.1016/S0304-3940(03)00160-5View ArticlePubMedGoogle Scholar
- Ramachandran VS: Plasticity and functional recovery in neurology. Clin Med 2005, 5: 368-373. 10.7861/clinmedicine.5-4-368View ArticleGoogle Scholar
- Chan BL, Witt R, Charrow AP, Magee A, Howard R, Pasquina PF, Heilman KM, Tsao JW: Mirror therapy for phantom limb pain. N Engl J Med 2007, 357: 2206-2007. 10.1056/NEJMc071927View ArticlePubMedGoogle Scholar
- Brodie EE, Whyte A, Niven CA: Analgesia through the looking-glass? A randomized controlled trial investigating the effect of viewing a 'virtual' limb upon phantom limb pain, sensation and movement. Eur J Pain 2007,11(4):428-436. 10.1016/j.ejpain.2006.06.002View ArticlePubMedGoogle Scholar
- Altschuler EL, Wisdom SB, Stone L, Foster C, Galasko D, Llewellyn DM, Ramachandran VS: Rehabilitation of hemiparesis after stroke with a mirror. Lancet 1999, 353: 2035-2036. 10.1016/S0140-6736(99)00920-4View ArticlePubMedGoogle Scholar
- Yavuzer G, Selles R, Sezer N, Sütbeyaz S, Bussmann JB, Köseoğlu F, Atay MB, Stam HJ: Mirror therapy improves hand function in subacute stroke: a randomized controlled trial. Arch Phys Med Rehabil 2008,89(3):393-398. 10.1016/j.apmr.2007.08.162View ArticlePubMedGoogle Scholar
- Ezendam D, Bongers RM, Jannink MJ: Systematic review of the effectiveness of mirror therapy in upper extremity function. Disabil Rehabil 2009,31(26):2135-49. 10.3109/09638280902887768View ArticlePubMedGoogle Scholar
- Feltham MG, Ledebt A, Bennett SJ, Deconinck FJ, Verheul MH, Savelsbergh GJ: The "mirror box" illusion: effect of visual information on bimanual coordination in children with spastic hemiparetic cerebral palsy. Motor Control 2010,14(1):68-82.PubMedGoogle Scholar
- Moseley GL: Using visual illusion to reduce at-level neuropathic pain in paraplegia. Pain 2007, 130: 294-298. 10.1016/j.pain.2007.01.007View ArticlePubMedGoogle Scholar
- Mercier C, Sirigu A: Training with virtual visual feedback to alleviate phantom limb pain. Neurorehabil Neural Repair 2009,23(6):587-594. 10.1177/1545968308328717View ArticlePubMedGoogle Scholar
- Murray CD, Patchick E, Pettifer S, Caillette F, Howard T: Immersive virtual reality as a rehabilitative technology for phantom limb experience: a protocol. Cyberpsychol Behav 2006,9(2):167-170. 10.1089/cpb.2006.9.167View ArticlePubMedGoogle Scholar
- O'Neill K, de Paor A, MacLachlan M, McDarby G: An Investigation into the Performance of a Virtual Mirror Box for the Treatment of Phantom Limb Pain in Amputees using Augmented Reality Technology. In Human-Computer-Interaction International. Conference Proceedings. London: Lawrence Erlbaum Associates Inc; 2003.Google Scholar
- Whyte AS, Niven CA: Variation in phantom limb pain: results of a diary study. J Pain Symptom Manage 2001,22(5):947-953. 10.1016/S0885-3924(01)00356-6View ArticlePubMedGoogle Scholar
- van den Bogert AJ, Geijtenbeek T, Even-Zohar O, Steenbrink F, Hardin EC: A real-time system for biomechanical analysis of human movement and muscle function. Med Biol Eng Comput in pressGoogle Scholar
- Geijtenbeek T, Steenbrink F, Otten E, Even-Zohar O: D-Flow: Immersive Virtual Reality and Real-Time Feedback for Rehabilitation. In Proceedings of the 10th ACM International Conference on Virtual Reality Continuum and Its Applications in Industry, 11-12 December 2011. New York: ACM; 2011:201-208.Google Scholar
- Hreljac A, Marshall RN: Algorithms to determine event timing during normal walking using kinematic data. J Biomech 2000, 33: 783-786. 10.1016/S0021-9290(00)00014-2View ArticlePubMedGoogle Scholar
- O’Connor CM, Thorpe SK, O’Malley MJ, Vaughan CL: Automatic detection of gait events using kinematic data. Gait Posture 2007, 25: 469-474. 10.1016/j.gaitpost.2006.05.016View ArticlePubMedGoogle Scholar
- Miller A: Gait event detection using a multilayer neural network. Gait Posture 2009, 29: 542-545. 10.1016/j.gaitpost.2008.12.003View ArticlePubMedGoogle Scholar
- De Asha AR, Robinson MA, Barton GJ: A marker based kinematic method of identifying initial contact during gait. Gait Posture 2012,36(3):650-652. 10.1016/j.gaitpost.2012.04.016View ArticlePubMedGoogle Scholar
- Zeni JA Jr, Richards JG, Higginson JS: Two simple methods for determining gait events during treadmill and overground walking using kinematic data. Gait Posture 2008, 27: 710-714. 10.1016/j.gaitpost.2007.07.007View ArticlePubMedPubMed CentralGoogle Scholar
- Levine D, Richards J, Whittle MW: Whittle’s Gait Analysis. Churchill Livingstone: Elsevier; 2012.Google Scholar
- Barton GJ, De Asha AR, Geijtenbeek T, Robinson MA: Development of a Virtual Mirror Box for spatial and temporal manipulation of visual feedback on body movement during gait: A technical evaluation [abstract]. Gait Posture 2011,36(S1):76.Google Scholar
- Ustinova KI, Perkins J, Szostakowski L, Tamkei LS, Leonard WA: Effect of viewing angle on arm reaching while standing in a virtual environment: Potential for virtual rehabilitation. Acta Psychologica 2010, 133: 180-190. 10.1016/j.actpsy.2009.11.006View ArticlePubMedGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.