Learning real-life cognitive abilities in a novel 360°-virtual reality supermarket: a neuropsychological study of healthy participants and patients with epilepsy
© Grewe et al.; licensee BioMed Central Ltd. 2013
Received: 18 August 2011
Accepted: 25 March 2013
Published: 23 April 2013
To increase the ecological validity of neuropsychological instruments the use of virtual reality (VR) applications can be considered as an effective tool in the field of cognitive neurorehabilitation. Despite the growing use of VR programs, only few studies have considered the application of everyday activities like shopping or travelling in VR training devices.
We developed a novel 360°- VR supermarket, which is displayed on a circular arrangement of 8 touch-screens – the “OctaVis”. In this setting, healthy human adults had to memorize an auditorily presented shopping list (list A) and subsequently buy all remembered products of this list in the VR supermarket. This procedure was accomplished on three consecutive days. On day four, a new shopping list (list B) was introduced and participants had to memorize and buy only products of this list. On day five, participants had to buy all remembered items of list A again, but without new presentation of list A. Additionally, we obtained measures of participants’ presence, immersion and figural-spatial memory abilities. We also tested a sample of patients with focal epilepsy with an extended version of our shopping task, which consisted of eight days of training.
We observed a comprehensive and stable effect of learning for the number of correct products, the required time for shopping, and the length of movement trajectories in the VR supermarket in the course of the training program. Task performance was significantly correlated with participants’ figural-spatial memory abilities and subjective level of immersion into the VR.
Learning effects in our paradigm extend beyond mere verbal learning of the shopping list as the data show evidence for multi-layered learning (at least visual-spatial, strategic, and verbal) on concordant measures. Importantly, learning also correlated with measures of figural-spatial memory and the degree of immersion into the VR. We propose that cognitive training with the VR supermarket program in the OctaVis will be efficient for the assessment and training of real-life cognitive abilities in healthy subjects and patients with epilepsy. It is most likely that our findings will also apply for patients with cognitive disabilities resulting from other neurological and psychiatric syndromes.
The lack of ecological instruments in the field of clinical neuropsychology has been criticized frequently [1–4]. This criticism is particularly relevant in the field of complex higher cognitive functions such as (autobiographical) episodic memory [5–7] and executive functions . Among a number of approaches to increase ecological validity , the use of Virtual Reality (VR) has been increasingly considered to allow for an ecologically valid assessment of everyday cognitive functions [10, 11]. During the last two decades, a growing number of studies in the field of clinical neuropsychology used VRs for assessment and intervention purposes [for reviews see [12, 13]. The VR technique allows for both control of experimental manipulations (e.g., different levels of complexity) and precise measures of subjects’ responses  within relatively natural, immersive settings [15, 16].
As cognitive functions are known to be altered by age , there is a considerable need for rehabilitation programs in our aging society. In rehabilitation settings, shopping can be considered as one of the most important activities to maintain and/or regain elderly people’s and neurological and psychiatric patients’ independent everyday life functioning [18–20]. In spite of their high relevance to successful rehabilitation for neurological and psychiatric patients, only few of these everyday activities, including shopping, have been implemented in computer-assisted training programs and VR applications . In an early report, Cromby et al.  investigated real life transfer of cognitive tasks learned in a virtual supermarket or other VR scenarios in students with learning difficulties using a two-week training intervention. In the experimental group, participants were presented with a shopping task in a virtual supermarket, while the control group was trained inside various VRs (e.g., a virtual house, a virtual city). The authors reported that the experimental group needed less time and bought more correct items in a subsequent shopping task in a real supermarket in comparison to the control group. Lee et al.  conducted a five day training in a virtual supermarket presented via a head-mounted display. On a descriptive statistical level, they could show effects of learning during the course of the five day learning program, but they did not report any inferential statistics. Rand, Weiss and Katz  developed a virtual version of the Multi Errands Test [MET; , a standardized test to assess and train multitasking behavior in a real shopping mall, by adapting the MET into a virtual mall presented via a video capture system. In the MET, the participant has to run several different given errands, which require a strategic planning because of restrictions and rules to be followed (e.g., order of the errands, time restrictions, different opening hours). Using this VR-MET, the authors investigated effects of ten training sessions in four stroke patients. They observed a decrease of rule breaks and non-efficient strategies in both the traditional and the virtual MET. Klinger et al. developed a virtual supermarket task presented on a 17” LCD screen to assess executive functions in different patient groups. Action planning was measured during a shopping task, which required the participants to plan his/her shopping according to a given shopping list . Using this paradigm, the authors successfully showed in a number of studies impaired and spared aspects of executive functions in patients with mild cognitive impairment , stroke , Parkinson’s disease  and schizophrenia . However, they did not report effects of training using their shopping task.
Available data demonstrate that VR supermarkets can efficiently be applied for the assessment and training of cognitive functions. Previous studies applied virtual supermarkets with relatively low resemblance to real life supermarkets  and the technical presentation was rather simple (e.g., use of small LCD-screens). Also, previous work has mainly focused on case study designs and descriptive statistics. Most importantly, VR supermarkets were used only for neuropsychological assessment, but not for training purposes.
Based on the current state-of-the-art of VR applications in neuropsychological diagnosis and rehabilitation, we developed a novel 360°- virtual reality supermarket displayed on a circular arrangement of 8 touch-screens – the “OctaVis” . This 360°- apparatus allows for intuitive real world-like movements, as the participant is able to turn around and rotate freely and interact with virtual items by real world-like movements (e.g., reaching out with the arm and hand for articles). It offers the opportunity to display high-resolution 3D-graphics at 360°- field of view, which can be understood as an advantage over the aforementioned studies. The main aim of this study was to evaluate the efficacy of a new 360°- VR supermarket task for the ecologically valid assessment and training of cognitive functions. Therefore, we analyzed participants’ performance and level of presence in our new 360°- OctaVis, using a virtual shopping task. We propose that our technically more advanced presentation of the VR may induce stronger feelings of immersion and this will enhance the efficacy of training within the VR [30, 31]. Our assumption is based on previous research showing positive effects of display size and participants’ field of view on the subjective feeling of immersion [32, 33]. In particular, we hypothesize that training in our 360°-VR supermarket will lead to substantial learning in remembering and finding articles of a shopping list. Furthermore, learning of this list will not be interrupted by the introduction of a new, interfering shopping list. Finally, we assume participants’ level of immersion into the VR and feasibility of our task to predict levels of learning in our task. To address the issue whether training effects are comparable in a clinical sample, we also included a small sample of patients with focal epilepsy in our study. Due to the high incidence of memory impairments in this clinical group [34, 35], patients were considered to have a special benefit from our VR training program of memory functions in an everyday-like context.
Participants were 19 healthy university students (5♂, 14♀). Mean age was 23 ± 3.45 (range 19 to 28 years). To ensure a homogeneous sample with average general cognitive functioning, we assessed participants’ IQ. Mean IQ was 109.9 ± 8.58. Participants’ medical history was assessed via a self-report questionnaire and a subsequent interview for detailed medical anamnesis. Participants with lifetime head injuries, severe medical illness, medication affecting the central nervous system, psychiatric diseases, and neurological diseases were excluded from the study. Also, participants with current consumption of illegal drugs or alcohol abuse were excluded.
We also examined an additional small sample of patients with focal epilepsy (n = 5). Mean age of patients was 35.04 ± 8.08 (range 25 to 47), mean IQ was 104 ± 9.23. Additional file 1 provides a detailed overview of the patients’ clinical and demographic data. In the clinical sample, neuropharmacological medication of epilepsy was not generally an exclusion criterion. However, patients who were treated with substances causing memory disturbances were excluded from the study.
Written informed consent was obtained from all subjects prior to participation, and the local ethics committee approved the study.
For the epilepsy sample, we administered an extended eight-day version of our supermarket training program instead of the five-day training applied in healthy adults. This modification was implemented since even healthy volunteers were unable to remember and buy all 20 items of the target list on day 3, that is, the day before the interference list was presented. The absence of ceiling effects in normal subjects indicates a rather high difficulty of our VR shopping paradigm, which is sensitive for the differentiation between high and low performers in healthy young adults. It is therefore likely that a three-day learning of the target list may be too short for learning both target list items and orientation in the VR supermarket in a clinical sample. The extended 8-day training thus included 6 days of training of the target list (instead of 3 days) before the interference list was presented. Except the prolonged learning phase of the target list, the study design was the same as in the five- day program. Figure 1B shows a schematic overview of the extended training program applied to patients with epilepsy.
None of the articles following directly one after the other in lists A or B were placed next to each other in the virtual supermarket. For acoustic stimulation, we used a low background sound, which before had been recorded in a real supermarket and consisted of sounds, which can be typically found in a supermarket (e.g., customers passing by, announcements, customers’ conversations). Sound files were provided via four stereo speakers, which were installed on top of the OctaVis (Genelec, Iisalmi, Finland, http://www.genelec.com, Figure 2). There were no virtual customers or other persons in the supermarket.
Presence and immersion
As the emergence of feelings of presence can be considered as a result of individual characteristics of a person and/or properties of a particular virtual environment , we obtained two different measures of immersion:
To investigate the participants’ perceived level of presence during the shopping situation in the virtual supermarket (state measure of immersion), we applied the Presence Questionnaire [PQ;  after the first day of training (day 1) and, again, after the last day of training (day 5). The PQ is a self rating questionnaire that assesses participants’ momentary level of presence, or the feeling of being immersed into a particular VR. Participants’ answers can be given on a 7-point Likert-scale. It is constructed to assess a multi-faceted construct of presence. In a principal components factor analysis, Witmer et al.  revealed that four PQ-subscales account best for the total variance data. These are “involvement”, “sensory fidelity”, “adaptation/immersion”, and “interface quality”.
In addition, before beginning the VR training, participants completed the Immersive Tendencies Questionnaire [ITQ; . The ITQ is a self-rating questionnaire assessing an individual’s general tendency to get immersed into a virtual environment (trait measure of immersion). Trait measures of immersion can be highly different between individuals. In contrast to the PQ, the ITQ thus does not measure a current state of presence at a particular point in time. Therefore we applied the ITQ only at one time point. The items of the ITQ each comprise a 7-point Likert-scale and comprise three subscales: “involvement”, “focus” and “games”.
To control for the occurrence of cybersickness, we also asked the participants whether they experienced symptoms of vertigo as an indicator of feasible cybersickness.
We also examined usability of both the task and the technical apparatus using a self-constructed explorative questionnaire, which included four items belonging to the “task”-subscale and four items belonging to the “technique”-subscale (Additional file 2). The questionnaire was designed in style of the PQ and was given to the participants directly after the PQ on day 1 and day 5.
Finally, we assessed participants’ figural memory with the Rey–Osterrieth Figure [ROF;  before the beginning of our program: Participants first had to copy a complex geometric figure (measuring visuo-constructive and planning abilities). Then, three and 30 minutes later, they were asked to freely recall this figure by drawing it again from memory.
For behavioral measures of performance in the virtual supermarket, we considered the time required by the participants to buy all the shopping items they remembered (“time”) as well as the number of correctly picked items from the respective list (“correct products”) the adjusted number of correctly picked items from the respective list (“product-score”; i.e., number of correct items minus false positives minus repetitions) and the length of movement trajectories (“LMT”). The LMT is given in meters and refers to the distance travelled in the VR supermarket by each participant to perform the task on each day. LMT accordingly represents the length of movement trajectories the participant would have traveled in a real 25 × 25 m supermarket.
Since we mainly focused on the analysis of performance of a healthy control group in our novel VR training program, we were particularly interested in behavioral raw measures on each single day of training. Our approach of data analysis allowed us to assess in detail how performance changed between two single trials/days and on which stages of training significant changes of performance occurred. For the same reason, we were also interested in correlations between performance on single trials/days in our VR program and scores on standard neuropsychological tests of figural memory as well as measures of immersion.
All statistical analyses were conducted using SPSS 17 (SPSS Inc., Chicago, Illinois). The general significance level was set to α = .05. To check for the assumption of normally distributed variables, we used the Kolmogorov-Smirnov-Test (α = .20). Associations between the measures of learning and the results of the applied questionnaires were calculated by using Pearson’s correlation coefficient, r. Effects of learning in the course of the training were analyzed using matched pairs t-tests in the case of a comparison between two single days. The comparison between days 3 and 5 was calculated to test for potential interfering effects of the distraction list B on free recall of the target list A. In cases where comparisons across more than two days were calculated, we used repeated measures ANOVAs. If the repeated measures ANOVAs revealed significant differences between trials, post-hoc comparisons for the differences between each single pair of days were conducted using dependent t-tests. Multiple post-hoc-comparisons were corrected using Bonferroni adjustment. The Bonferroni adjusted significance level for the post-hoc-comparisons was α = .0083.
Due to the small sample size (n = 5), nonparametric tests were chosen for the analyses including the clinical group. Thus, associations between two variables were calculated using Spearman’s rho, ρ s . Comparisons between single trials were analyzed using Wilcoxon signed rank tests.
The Kolmogorov-Smirnov-Tests revealed non- significant results for all variables used in our analysis, thus indicating normal distribution of the variables.
Efforts of learning
Descriptive and inferential statistics of the correct products
day 1 vs. 2
day 2 vs. 3
day 3 vs. 4
day 4 vs. 5
day 1 vs. 4
day 3 vs. 5
Descriptive and inferential statistics of the length of movement trajectories
day 1 vs. 2
day 2 vs. 3
day 3 vs. 4
day 4 vs. 5
day 1 vs. 4
day 3 vs. 5
Descriptive and inferential statistics of the required time
day 1 vs. 2
day 2 vs. 3
day 3 vs. 4
day 4 vs. 5
day 1 vs. 4
day 3 vs. 5
Immersion and presence
Neither the total score of the ITQ nor one of the ITQ- subscales was correlated with the number of correct products, the product-score, the LMT or the required time.
The PQ total score increased from day 1 (M = 137.26, SD = 18.62) to day 5 (M = 143.89, SD = 13.47), but there was only a trend for a significant difference (t = −1.89, p = .075, d = .433). The PQ-subscale “immersion/adaptation” increased from day 1 (M = 40.74, SD = 6.62) to day 5 (M = 45.53, SD = 4.91; t = −2.76, p = .013, d = .632).
Person correlations between VR performance and measures of immersion, figural memory and task feasibility
ROF 30’ recall
Participants’ mean T-Score in the immediate recall of the ROF after three minutes was 55.58 (SD = 10.43), the mean delayed recall after 30 minutes was 53.16 (SD =10.63).
The task-subscale of our explorative questionnaire significantly increased from day 1 (M = 17.53/28, SD = 2.39) to day 5 (M = 19.32/28, SD = 2.08; t = −4.04, p < .001, d = .927). The technique-subscale of our explorative questionnaire significantly increased from day 1 (M = 22.58/28, SD = 3.98) to day 5 (M = 27.74/28, SD = 3.02; t = −4.19, p < .001, d = .963).
The task-subscale was significantly correlated was the LMT on day 1 (r = −.596, p = .009) and day 4 (r = −.631, p = .005) and the number of correct products on day 1 (r = .476, p = .046). Table 4 summarizes the main results of the correlational analyses.
On day 1, nine of nineteen participants reported signs of vertigo, on day 5 only two of nineteen participants reported signs of vertigo. None of the participants cancelled participation in our study due to vertigo.
After the last training trial on day 5, participants were asked whether they used any kind of strategy to remember the items of the lists during the experiment via a self-constructed questionnaire. In particular, they were firstly asked whether they had applied any kind of strategy to remember the items of the shopping lists. If the application of any strategy was affirmed, participants had to describe and write down the applied strategy by their own words. Multiple answers were allowed. The following strategies were mentioned by the participants: “Dual-coding“ (i.e., the association of verbal information with visual-spatial information [design of each product and/or its localization in the VR supermarket]; 41%), “semantic clustering “(17%), “recognition” (6%), “verbal rehearsal” (6%), “serial clustering” (6%), “body-part method” (i.e., association of items with parts of the body; 6%), “counting of products” (6%), other (12%).
Results of the clinical sample
Because of the small sample size, learning efforts across the eight day training program could not be calculated by repeated measurement ANOVAs. However, Figure 5 (right side) provides an explorative overview of the learning efforts (i.e., correct products, LMT, and time) of the clinical group. Concerning the main hypotheses, participants bought more items on day 7 (median = 12, range = 4) in comparison to day 1 (median = 9, range = 5; Z = 2.04, p = .042; Figure 5A, right side). The required time for shopping decreased from day 6 (median = 235, range = 575) to day 8 (median = 187, range = 64; Z = 2.02, p = .043; Figure 5C, right side). The LMT and the number of correct products did not differ between day 6 and day 8. There was a trend for a correlation between the number of correctly bought products on day 8 and the immediate (ρ s = .872, p = .054) and the delayed recall (ρ s = .872, p = .054; Figure 6B) of the ROF.
In this study, we investigated the feasibility of a VR supermarket application presented in the OctaVis, a novel 360°- VR apparatus, for the assessment and training of neuropsychological functions in a sample of healthy young adults and a small clinical sample of patients with focal epilepsy. Healthy participants learned a list of shopping articles and subsequently bought them in a virtual supermarket on three consecutive days. On day 4, participants learned and bought items of a new, distractive list. Finally, on day 5, participants had to buy only the items of the target list, but without a new presentation of this list. Results show increasing levels of learning throughout the task as well as high levels of subjects’ immersion in the VR. Moreover, performance in our task was significantly correlated with a measure of figural-spatial memory. The time needed for completion of the shopping task was significantly longer on day 1 than on day 4 (application of the distractive list B), indicating that visual-spatial familiarity with the structure of the supermarket decreased the immediate distracting effect of list B. Also, we did not observe a decrease of performance from day 3 to day 5, thus indicating that learning was not significantly interrupted by the introduction of a new interfering list. Importantly, immersion was positively correlated with the number of correctly bought products, suggesting that immersive feelings may enhance cognitive performance in everyday-like neuropsychological tasks.
In the course of training in the OctaVis, there was a considerable improvement of cognitive performance across training sessions. This effect of learning was observed for several concurrent measures: Participants successively needed less time and shorter movement paths to accomplish the task and bought a higher number of correct products, which was even valid when correcting for incorrect products and repetitions. Moreover, performance of the ROF was positively correlated with the number of correctly bought products. Finally, our main findings could be replicated in our clinical sample showing (a) comprehensive and stable learning, (b) no negative effect on performance after the introduction of the interference list, and (c) and association of measures of VR performance and the ROF in our patients with epilepsy.
We suppose that these comprehensive results of learning do not only reflect verbal learning as a consequence of the repeatedly presented shopping-list. Rather, improvement on the different but converging measures support the view that learning occurred mainly on verbal, visual-spatial, executive, and familiarity levels . Although our data do not directly allow for conclusions on multi-layered learning, we propose that successful learning in the VR shopping task depends on the integration of at least verbal and visual-spatial modalities of learning, that is, a form of dual-coding. Moreover, the task requires executive abilities, in particular, visual-spatial planning strategies. This view is supported by the correlations between the ROF and learning scores in our VR task. The ROF represents visual-spatial memory functions, which are mostly independent of verbal abilities . Besides visual-spatial memory abilities, the ROF also requires planning and structuring abilities and has correspondingly been used as a measure of planning and organization in previous studies [39–45]. We therefore propose that the correlation between our task and the ROF might represent the task’s requirement of at least verbal learning and non-verbal figural learning, as well as executive abilities, thus supporting our argument of multi-layered learning in the VR supermarket. Furthermore, our idea of a multi-layered learning process is also supported by the strategies participants used to accomplish the task. The use of dual coding, which implies the dual coupling of words of the shopping list with the visual representation of the respective product in the VR supermarket, was the memory strategy that was most frequently reported by our participants. It is most likely that the VR shopping task also makes demands on visual-spatial orientation and way-finding. However, our data do not contribute to this issue such that this assumption remains speculative. In our study, we did not aim at offering a full psychometric validation of our novel instrument. Furthermore, procedural learning and habituation to both technical control of the shopping task and the visual spatial structure of the VR environment may also have contributed to efficient learning. Eventually, our idea of a multi-layered learning that took place in our VR task goes in line with the concept of multiple memory systems that represent different subtypes of memory and learning processes [6, 46, 47].
Interestingly, the data show evidence for further learning from day 1 (first entering the supermarket) to day 4 (entering the familiar supermarket after having heard a new shopping list) for the time required for the shopping task and the LMT, but not for the product-score. This decrease of time and LMT on day 4 (relative to day 1) indicates that figural-spatial learning of the supermarket’s routes and layout occurred, which may be relatively independent of verbal memory of list A articles and their localization in the supermarket. This dissociation between verbal and figural-spatial learning further confirms our aforementioned idea of multi-layered learning. In particular, we suppose that participants created a cognitive map [48, 49] of the virtual supermarket. Previous studies are in accordance with this assumption. The idea that generation of cognitive maps could be enhanced by learning in a VR , was supported by Tong et al.  who found that active performance in a VR-pathfinding task enhanced the accuracy in a subsequent drawing of a cognitive map of the landmarks in the VR. Results thus suggest that visual-spatial learning might have played a central role for task performance in our task besides verbal learning of the products. This assumption is in good accordance with the frequent application of dual-coding strategies reported by the volunteers (i.e., verbal and visual-spatial representation).
Insofar, our results are in line with a study of Brooks et al. , who found a dissociation between object and spatial learning in a VR. While the spatial layout of the VR could be recalled more accurately by participants who actively (vs. passively) navigated through the VR, the mere learning of objects placed in a VR was independent of (active vs. passive) navigation in the VR . Moreover, in a study of patients with traumatic brain injury, Matheis et al.  found a dissociation between patients’ impaired list learning and spared visual memory performance.
We propose that learning of the layout of the VR supermarket took place on the first two days of training since the data show the highest increases of performance from day 1 to day 2. Moreover, the interference list did not affect performance in our VR shopping paradigm. Thus, there were no significant differences between all measures of performance on day 3 and day 5 (i.e., product-score, LMT, and time). Hence, our initial hypothesis that learning will not be interrupted by the introduction of a new shopping list was corroborated by the data, that is, shopping performance of the target list was comparable on days 3 and 5, although we inserted new learning materials in between these trials. It is therefore reasonable to assume that visual-spatial learning supported the emergence of a multi-layered representation of the shopping articles included in the target list. Importantly, we replicated this result in our preliminary study of the small clinical sample of patients with focal epilepsy. This argues in favor of a stable and robust representation of learned multi-layered information after an initial learning phase of the visual-spatial layout of the VR supermarket in both healthy controls and clinical samples.
For our present data, we suppose that interaction with the highly immersive 360°- VR OctaVis may have prompted multi-modal learning. There is considerable evidence from studies in children and adults that multi-modal learning is more efficient, deep and stable over time than unimodal learning [54–58]. Multi-modal learning may also enhance the feeling of presence and learning in VRs [59, 60]. In addition, combined training of multiple cognitive functions could be shown to be an efficient strategy for rehabilitation of memory problems . It is reasonable to assume that both the realistic layout of the supermarket and the real-life like interactive movements (e.g., turning around, reaching out with the arm and hand for an article) may have supported the integration of visual-spatial and motor learning, and therefore facilitated the formation of episodic in contrast to mere semantic memory contents , which makes our task a more precise measure of real-life cognitive performance. Most likely, immersion in the VR is a key player in the emergence of efficient multi-modal learning since it builds up the basis for real-life like integration of visual perception, motor-action and visual-spatial memory [52, 63].
With regard to immersion, we observed increasing levels of subjective immersive feelings during the course of our training program in the 360°- VR supermarket. Insofar, our results are in good agreement with Lee et al.  who could also show participants’ increasing levels of subjective immersion into the VR in the course of a five day intervention. In our study, it is likely that participants became more and more familiar with the supermarket and the task and could thus successively immerse better into the VR during the course of the training intervention [64, 65]. Increase of immersion may well be related to learning success. The correlations between measures of learning (i.e., product-score, time) and intensity of immersion, which both increased during the course of the task further supports this view. Our results provide first evidence that learning success depends at least in part on subjective immersive feelings. Eventually, our results are in accordance with the findings of previous studies showing that intense feelings of immersion may be associated with higher levels of task performance in a more general sense  and better treatment success in psychotherapy settings where VR is used to cure different forms of phobias .
As in previous VR studies using the PQ [22, 66–68] we observed relatively high levels of immersion. On the one hand, this further underlines our apparatus’ technical feasibility for presenting highly immersive VR. We particularly assume that participants’ immersion was at least in parts enhanced by our novel apparatus with its specially designed 360°- view, the multi-sensory (i.e. visual, motor and auditory) design and the intuitive and interactive control. This is in line with several former research showing positive effects of field-of-view size [32, 33], multi-modal integration  and active control on immersion . On the other hand, it can be also supposed that the high levels of immersion found in our study will beneficially contribute to feasible transfer effects as immersion is thought to be a critical factor to enhance transfer from VR to real-life situations .
In contrast to these correlational associations between momentary presence and immersion measured with the PQ and task performance in the OctaVis, we did not find any association between performance in the OctaVis and the scales of the ITQ. Thus, a subject’s individual tendency for immersion was not found to be a critical factor for our VR training paradigm. We therefore conclude that performance in our paradigm mainly depends on the momentary level of immersion into the VR (as assessed with the PQ), rather than on a subject’s general personal trait or capability to get involved or immersed (as assessed with the ITQ). This aspect is of high importance for a future routine clinical application of VR scenarios in the OctaVis since the efficiency of training in the OctaVis should accordingly be independent of a person’s individual trait to get immersed into VR. This distinction allows for successful application of our program to a wider range of participants and does not restrict the application to participants with computer experience. Certainly, a further investigation examining a sample with a wider range of ages would add important information to this issue.
As it is important to use generally valid, “cross-media” measures of presence as well as specific measures that fit the proper and special technical features of the VR and its control devices , we also looked at the way participants subjectively got along with our particular task (represented by the task-subscale) and the technical apparatus (represented by the technique-subscale). We observed high scores in both subscales, which even increased significantly in the course of our program and were correlated with measures of learning in the VR task, thus indicating both an easy-to-handle control of our technical devices as a training program that seems intuitive and easy to handle. However, our questionnaire’s design was of explorative nature and scores may not be interpreted in terms of established measurements.
In summary, our findings provide preliminary evidence that our novel VR supermarket paradigm presented in the OctaVis may efficiently be applied for the assessment and training of real-life cognitive functions in healthy subjects and patients with focal epilepsy. However, we acknowledge some limitations and caveats interpreting our study. First, we did not control for each of the proposed levels of learning in particular. For example, we could not assess a verbal list learning paradigm as this would have interfered with the learning in our virtual paradigm. Although our study does not claim to offer a full-scale validation study, the application of additional specifically related “traditional” neuropsychological tests (e.g., of verbal memory) would presumably have added important information about single cognitive processes involved in performance in our novel VR paradigm. In particular, a labyrinth task needs to be included in future studies to assess the roles of visual-spatial orientation and way-finding in the VR shopping task. Moreover, list learning paradigms like the California Verbal Learning Task  or the Verbal Learning- und Memory Test [German adaptation of the Auditory Verbal Learning Test; , which comprise the most established measures of verbal memory , might offer important information on the role of verbal memory in the VR shopping task. Thus, the question whether both verbal and visual memory processes might have played a role could have been adressed more directly. In future experiments, we will hence offer additional measures of verbal memory performance. Second, our results should be replicated with a sample of older participants to evaluate generalization of our results to a group of participants with less computer experience. We did not directly address the participants’ motivation during shopping in the VR supermarket on each day of training. It is reasonable to assume that besides cognitive capacity, different levels of motivation and effort could have had an influence on the participants’ performance in the VR supermarket. Moreover, participants’ experience of cybersickness needs to be considered as another constraint of our study. Using a questionnaire on experiences of cybersickness we aimed at identifying sources of cybersickness during training in the OctaVis. Most participants reported a flicker in their peripheral field of view as an eliciting factor of nausea. This flicker is mainly technically related to fast movements in the VR, a relatively low frame-rate of the VR system (i.e. a low speed of re-generation of the VR environment), and the large field of view . In our current studies of training in the OctaVis, we have accelerated the frame-rate and limited the maximum speed of movement inside the VR to reduce symptoms of cybersickness elicited by these factors. These changes of technical parameters resulted in a considerable reduction of reports of cybersickness. Finally, our task’s psychometric properties must be further evaluated. Therefore, we will apply the OctaVis to different populations to examine its validity in laboratory and further clinical settings. We currently elaborate different forms of feedback, which can be suitable for different patient groups. In parallel, we compare our VR program with already established training paradigms.
Based on diverse concordant measures of visual-spatial, strategic, and verbal cognition, we showed evidence for a comprehensive and multi-layered learning success in the course of the training. Correlations between measures of multi-layered learning and scores on classical neuropsychological tests of visual-spatial cognition corroborate the view that our findings do not only depend on verbal learning of the shopping list. Moreover, we were able to demonstrate the feasibility of the VR supermarket paradigm presented in the OctaVis by high scores on both an established presence questionnaire and a self-constructed questionnaire, which was specifically related to our technical device. Importantly, we also provide strong evidence of a relation between the level of immersion and task performance. We conclude that we have developed and tested a novel 360°- VR environment, which demands real world-like visual-spatial and motor actions and thus allows for the training of the respective cognitive abilities. Importantly, results of our sample of patients with focal epilepsy corroborate the main findings of our basic healthy sample, thus giving preliminary evidence of a replication of comparable effects in a clinical sample. We propose that our technical device and neuropsychological paradigm can efficiently be used for assessment and training of real life-based cognitive functions. Future studies providing both additional validation data and clinical evidence are needed to corroborate the efficiency of cognitive training interventions in the OctaVis in patients with different neurological and psychiatric diseases.
This work was supported by the European Fund of Regional Development (Europäischer Fonds für Regionale Entwicklung; EFRE) through the project "CITmed: Cognitive Interaction Technology for Medical Applications", and by the German Research Foundation through the Centers of Excellence DFG EXC 277 "Cognitive Interaction Technology" and DFG EXC 257 "NeuroCure". We thank our colleagues from the CITmed project, from CITEC at Bielefeld University, and from "rmh new media", for their scientific and technical support and helpful advice. We acknowledge the support for the publication fee by the German Research Foundation and the Open Access Publication Funds of Bielefeld University.
- Spooner DM, Pachana NA: Ecological validity in neuropsychological assessment: a case for greater consideration in research with neurologically intact populations. Arch Clin Neuropsychol 2006, 21: 327-337. 10.1016/j.acn.2006.04.004View ArticlePubMedGoogle Scholar
- Chaytor N, Schmitter-Edgecombe M: The ecological validity of neuropsychological tests: a review of the literature on everyday cognitive skills. Neuropsychol Rev 2003, 13: 181-197.View ArticlePubMedGoogle Scholar
- Deisinger K, Markowitsch HJ: Die Wirksamkeit von Gedächtnistrainings in der Behandlung von Gedächtnisstörungen. Psychol Rundsch 1991, 42: 55-65.Google Scholar
- Heinrichs RW: Current and emergent applications of neuropsychological assessment: Problems of validity and utility. Prof Psychol Res Pract 1990, 21: 171-176.View ArticleGoogle Scholar
- Piefke M, Weiss PH, Zilles K, Markowitsch HJ, Fink GR: Differential remoteness and emotional tone modulate the neural correlates of autobiographical memory. Brain 2003, 126: 650. 10.1093/brain/awg064View ArticlePubMedGoogle Scholar
- Markowitsch HJ, Piefke M: The functional neuroanatomy of learning and memory. Handbook Clin Neuropsychol 2010, 1: 793-810.View ArticleGoogle Scholar
- Piefke M, Fink GR: Recollections of one’s own past: The effects of aging and gender on the neural mechanisms of episodic autobiographical memory. Anat Embryol (Berl) 2005, 210: 497-512. 10.1007/s00429-005-0038-0View ArticleGoogle Scholar
- Onur ÖA, Piefke M, Lie C, Thiel CM, Fink GR: Modulatory Effects of Levodopa on Cognitive Control in Young but not in Older Subjects: A Pharmacological fMRI Study. J Cogn Neurosci 2011, 23: 2797-2810. 10.1162/jocn.2011.21603View ArticlePubMedGoogle Scholar
- Chaytor N, Schmitter-Edgecombe M, Burr R: Improving the ecological validity of executive functioning assessment. Arch Clin Neuropsychol 2006, 21: 217-227. 10.1016/j.acn.2005.12.002View ArticlePubMedGoogle Scholar
- Campbell Z, Zakzanis KK, Jovanovski D, Joordens S, Mraz R, Graham SJ: Utilizing virtual reality to improve the ecological validity of clinical neuropsychology: an FMRI case study elucidating the neural basis of planning by comparing the tower of London with a three-dimensional navigation task. Appl Neuropsychol 2009, 16: 295-306. 10.1080/09084280903297891View ArticlePubMedGoogle Scholar
- Parsons TD, Silva TM, Pair J, Rizzo AA: Virtual environment for assessment of neurocognitive functioning: virtual reality cognitive performance assessment test. In Medicine meets virtual reality 16: parallel, combinatorial, convergent: NextMed by design. Edited by: Westwood JD, Haluck RS, Hoffman HM, Mogel GT, Phillips R, Robb RA, Vosburgh KG. Amsterdam: Ios Press; 2008:351-356.Google Scholar
- Rose FD, Brooks BM, Rizzo AA: Virtual reality in brain damage rehabilitation: review. Cyberpsychol Behav 2005, 8: 241-262. discussion 263–271 10.1089/cpb.2005.8.241View ArticlePubMedGoogle Scholar
- Weiss PL, Kizony R, Feintuch U, Katz N: Virtual reality in neurorehabilitation. In Textbook of neural repair. Volume II. Edited by: Selzer ME, Clarke S, Cohen LG, Duncan PW, Gage FH. Cambridge: University Press; 2006:182-197. Medical NeurorehabilitationGoogle Scholar
- Rizzo AA, Schultheis MT, Kerns KA, Mateer CA: Analysis of assets for virtual reality applications in neuropsychology. Neuropsychol Rehabil 2004, 14: 207-239. 10.1080/09602010343000183View ArticleGoogle Scholar
- Castelnuovo G, Priore C, Liccione D, Cioffi G: Virtual Reality based tools for the rehabilitation of cognitive and executive functions: the V-STORE. PsychNology J 2003, 1: 310-325.Google Scholar
- Tarr MJ, Warren WH: Virtual reality in behavioral neuroscience and beyond. Nat Neurosci 2002,5(Suppl):1089-1092.View ArticlePubMedGoogle Scholar
- Piefke M, Onur ÖA, Fink GR: Aging related changes of neural mechanisms underlying visual-spatial working memory. Neurobiol Aging 2012,33(7):1284-1297. 10.1016/j.neurobiolaging.2010.10.014View ArticlePubMedGoogle Scholar
- Ponsford J, Harrington H, Olver J, Roper M: Evaluation of a community-based model of rehabilitation following traumatic brain injury. Neuropsychol Rehabil 2006, 16: 315-328. 10.1080/09602010500176534View ArticlePubMedGoogle Scholar
- Bennett TL: Neuropsychological evaluation in rehabilitation planning and evaluation of functional skills. Arch Clin Neuropsychol 2001, 16: 237-253.View ArticlePubMedGoogle Scholar
- Cromby JJ, Standen PJ, Newman JP, Tasker H: Successful transfer to the real world of skills practised in a virtual environment by students with severe learning difficulties. Maidenhead, UK: Proc 1st Euro Conf Disability, Virtual Reality & Assoc Tech; 1996:103-107.Google Scholar
- Parsons TD: Neuropsychological Assessment Using Virtual Environments: Enhanced Assessment Technology for Improved Ecological Validity. Advanced Computational Intelligence Paradigms in Healthcare 6 Virtual Reality in Psychotherapy, Rehabilitation, and Assessment. 2011, 271-289.Google Scholar
- Lee J, Ku J, Cho W, Hahn W, Kim I, Lee S, Kang Y, Kim D, Yu T, Wiederhold B: A virtual reality system for the assessment and rehabilitation of the activities of daily living. Cyberpsychol Behav 2003, 6: 383-388. 10.1089/109493103322278763View ArticlePubMedGoogle Scholar
- Rand D, Weiss PL, Katz N: Training multitasking in a virtual supermarket: a novel intervention after stroke. Am J Occup Ther 2009, 63: 535-542. 10.5014/ajot.63.5.535View ArticlePubMedGoogle Scholar
- Knight C, Alderman N, Burgess P: Development of a simplified version of the multiple errands test for use in hospital settings. Neuropsychol Rehabil 2002, 12: 231-255. 10.1080/09602010244000039View ArticleGoogle Scholar
- Werner P, Rabinowitz S, Klinger E, Korczyn A, Josman N: Use of the Virtual Action Planning Supermarket for the Diagnosis of Mild Cognitive Impairment. Dement Geriatr Cogn Disord 2009, 27: 301-309. 10.1159/000204915View ArticlePubMedGoogle Scholar
- Josman N, Hof E, Klinger E, Marie RM, Goldenberg K, Weiss PL, Kizony R: Performance within a virtual supermarket and its relationship to executive functions in post-stroke patients. In Virtual Rehabilitation, 2006 International Workshop. 2006, 106-109.Google Scholar
- Klinger E, Chemin I, Lebreton S, Marié R: Virtual Action Planning in Parkinson’s Disease: AControl Study. Cyberpsychol Behav 2006, 9: 342-347. 10.1089/cpb.2006.9.342View ArticlePubMedGoogle Scholar
- Josman N, Klinger E, Kizony R: Performance within the Virtual Action Planning Supermarket (VAP-S): An executive function profile of three different populations suffering from deficits in the central nervous system. Maia, Portugal: Proc 7th ICDVRAT; 2008.Google Scholar
- Dyck E, Schmidt H, Botsch M: OctaVis: A Simple and Efficient Multi-View Rendering System. GI VR/AR Workshop; 2010:1-8.Google Scholar
- Wiederhold BK, Wiederhold MD: Lessons Learned From 600 Virtual Reality Sessions. Cyberpsychol Behav 2000, 3: 393-400. 10.1089/10949310050078841View ArticleGoogle Scholar
- Witmer BG, Singer MJ: Measuring Presence in Virtual Environments: A Presence Questionnaire. Presence Teleop Virt 1998, 7: 225-240. 10.1162/105474698565686View ArticleGoogle Scholar
- Lin JWJ: Enhancement of user-experiences in immersive virtual environments that employ wide-field displays. University of Washington, Unpublished doctoral thesis Department of Industrial Engineering. 2004.Google Scholar
- Lombard M, Ditton TB, Grabe ME, Reich RD: The role of screen size in viewer responses to television fare. Comm Rep 1997, 10: 95-106. 10.1080/08934219709367663View ArticleGoogle Scholar
- Elger CE, Helmstaedter C, Kurthen M: Chronic epilepsy and cognition. Lancet Neurol 2004, 3: 663-672. 10.1016/S1474-4422(04)00906-8View ArticlePubMedGoogle Scholar
- Schefft BK, Dulay MF, Fargo JD, Szaflarski JP, Yeh HS, Privitera MD: The use of self-generation procedures facilitates verbal memory in individuals with seizure disorders. Epilepsy Behav 2008, 13: 162-168. 10.1016/j.yebeh.2008.01.012View ArticlePubMedGoogle Scholar
- Witmer BG, Jerome CJ, Singer MJ: The Factor Structure of the Presence Questionnaire. Presence Teleop Virt 2005, 14: 298-312. 10.1162/105474605323384654View ArticleGoogle Scholar
- Osterrieth P: Le test de copie d’une figure complexe; contribution à l’étude de la perception et de la mémoire. Arch Psychol 1944, 28: 215-285.Google Scholar
- Berry DTR, Allen RS, Schmitt FA: Rey-Osterrieth Complex Figure: Psychometric characteristics in a geriatric sample. Clin Neuropsychol 1991, 5: 143-153. 10.1080/13854049108403298View ArticleGoogle Scholar
- Park HS, Shin YW, Ha TH, Shin MS, Kim YY, Lee YH, Kwon JS: Effect of cognitive training focusing on organizational strategies in patients with obsessive‒compulsive disorder. Psychiatr Clin Neurosci 2006, 60: 718-726. 10.1111/j.1440-1819.2006.01587.xView ArticleGoogle Scholar
- Shin MS, Park SY, Park SR, Seol SH, Kwon JS: Clinical and empirical applications of the Rey-Osterrieth Complex Figure Test. Nat Protocol 2006, 1: 892-899. 10.1038/nprot.2006.115View ArticleGoogle Scholar
- Lezak MD: Neuropsychological assessment. 4th edition. Oxford: Oxford Univ. Press; 2004.Google Scholar
- Luzzi S, Pesallaccia M, Fabi K, Muti M, Viticchi G, Provinciali L, Piccirilli M: Non-verbal memory measured by Rey–Osterrieth Complex Figure B: normative data. Neurol Sci 2011, 32: 1081-1089. 10.1007/s10072-011-0641-1View ArticlePubMedGoogle Scholar
- Hamby SL, Wilkins JW, Barry NS: Organizational quality on the Rey-Osterrieth and Taylor Complex Figure Tests: A new scoring system. Psychol Assess 1993, 5: 27.View ArticleGoogle Scholar
- Binder LM: Constructional strategies on complex figure drawings after unilateral brain damage. J Clin Exp Neuropsychol 1982, 4: 51-58. 10.1080/01688638208401116View ArticleGoogle Scholar
- Shin M, Park S, Kim M, Lee Y, Ha T, Kwon J: Deficits of Organizational Strategy and Visual Memory in Obsessive-Compulsive Disorder. Neuropsychology 2004, 18: 665.View ArticlePubMedGoogle Scholar
- Markowitsch HJ, Staniloiu A: Amnesic disorders. Lancet 2012, 380: 1424-1440.View ArticleGoogle Scholar
- Squire LR: Declarative and nondeclarative memory: Multiple brain systems supporting learning and memory. J Cogn Neurosci 1992, 4: 232-243. 10.1162/jocn.19184.108.40.206View ArticlePubMedGoogle Scholar
- Tolman E: Cognitive maps in rats and men. Psychol Rev 1948, 55: 189.View ArticlePubMedGoogle Scholar
- Tversky B: Cognitive maps, cognitive collages, and spatial mental models. Spatial Information Theory A Theoretical Basis for GIS. 1993, 14-24.Google Scholar
- Johns C, Blake E: Cognitive maps in virtual environments: Facilitation of learning through the use of innate spatial abilities. ACM; 2001:125-129.Google Scholar
- Tong FH, Marlin SG, Frost BJ: Cognitive map formation in a 3D visual virtual world. IRIS/PRECARN; 1995.Google Scholar
- Brooks B, Attree E, Rose F, Clifford B, Leadbetter A: The specificity of memory enhancement during interaction with a virtual environment. Memory 1999, 7: 65-78. 10.1080/741943713View ArticlePubMedGoogle Scholar
- Matheis RJ, Schultheis MT, Rizzo AA: Learning and memory in a virtual office environment. In Proceedings of the 2nd International Workshop in Virtual Rehabilitation. Edited by: Burdea G, Thalmann D, Lewis JA. Piscataway: Rutgers University; 2003:48-54.Google Scholar
- Mayer RE: The promise of multimedia learning: using the same instructional design methods across different media. Learn Instr 2003, 13: 125-139. 10.1016/S0959-4752(02)00016-6View ArticleGoogle Scholar
- Collins HR: Pictorial encoding and testing impact recognition memory. Santa Barbara: University of California; 2008.Google Scholar
- Gellevij M, Van Der Meij H, De Jong T, Pieters J: Multimodal versus unimodal instruction in a complex learning context. J Exp Educ 2002, 70: 215-239. 10.1080/00220970209599507View ArticleGoogle Scholar
- Mayer RE: Multimedia aids to problem-solving transfer. Int J Educ Res 1999, 31: 611-623. 10.1016/S0883-0355(99)00027-0View ArticleGoogle Scholar
- Ngiam J, Khosla A, Kim M, Nam J, Lee H, Ng AY: Multimodal Deep Learning. 2010. InGoogle Scholar
- Davis ET, Scott K, Pair J, Hodges LF, Oliverio J: Can audio enhance visual perception and performance in a virtual environment. Human Factors and Ergonomics Society; 1999:1197-1201.Google Scholar
- Dinh HQ, Neff Walker CS, Kobayashi A, Hodges LF: Evaluating the importance of multi-sensory input on memory and the sense of presence in virtual environments. 1999, 222.Google Scholar
- Calabrese P, Markowitsch HJ: Recovery of mnestic functions after hypoxic brain damage. Int J Rehabil Health 1995, 1: 247-260. 10.1007/BF02214643View ArticleGoogle Scholar
- Tulving E, Markowitsch HJ: Episodic and declarative memory: role of the hippocampus. Hippocampus 1998, 8: 198-204. 10.1002/(SICI)1098-1063(1998)8:3<198::AID-HIPO2>3.0.CO;2-GView ArticlePubMedGoogle Scholar
- Péruch P, Vercher J, Gauthier G: Acquisition of spatial knowledge through visual exploration of simulated environments. Ecol Psychol 1995, 7: 1-20. 10.1207/s15326969eco0701_1View ArticleGoogle Scholar
- Hoffman HG, Prothero J, Wells MJ, Groen J: Virtual chess: Meaning enhances users’ sense of presence in virtual environments. Int J Hum Comput Interact 1998, 10: 251-263. 10.1207/s15327590ijhc1003_3View ArticleGoogle Scholar
- Lombard M, Ditton T: At the heart of it all: The concept of presence. J Comput-Mediat Commun 1997, 3: 0–0.View ArticleGoogle Scholar
- Lott A, Bisson E, Lajoie Y, McComas J, Sveistrup H: The effect of two types of virtual reality on voluntary center of pressure displacement. Cyberpsychol Behav 2003, 6: 477-485. 10.1089/109493103769710505View ArticlePubMedGoogle Scholar
- Rand D, Kizony R, Weiss PL: Virtual reality rehabilitation for all: Vivid GX versus Sony PlayStation II EyeToy. Citeseer; 2004:87-94.Google Scholar
- Usoh M, Catena E, Arman S, Slater M: Using presence questionnaires in reality. Presence Teleoperators Virt Environ 2000, 9: 497-503. 10.1162/105474600566989View ArticleGoogle Scholar
- Biocca F, Kim J, Choi Y: Visual touch in virtual environments: An exploratory study of presence, multimodal interfaces, and cross-modal sensory illusions. Presence Teleoperators Virt Environ 2001, 10: 247-265.View ArticleGoogle Scholar
- Péruch P, Wilson PN: Active versus passive learning and testing in a complex outside built environment. Cogn Process 2004, 5: 218-227. 10.1007/s10339-004-0027-xView ArticleGoogle Scholar
- Riva G, Mantovani F, Gaggioli A: Presence and rehabilitation: toward second-generation virtual reality applications in neuropsychology. J Neuroeng Rehabil 2004, 1: 9. 10.1186/1743-0003-1-9PubMed CentralView ArticlePubMedGoogle Scholar
- Lessiter J, Freeman J, Keogh E, Davidoff J: A cross-media presence questionnaire: The ITC-Sense of Presence Inventory. Presence: Teleoperators Virt Environ 2001, 10: 282-297. 10.1162/105474601300343612View ArticleGoogle Scholar
- Niemann H, Sturm W, Thöne-Otto A, Willmes K: CVLT–California verbal learning test–Deutsche adaptation. Frankfurt: Pearson; 2008.Google Scholar
- Helmstaedter C, Lendt M, Lux S: Verbaler Lern-und Merkfähigkeitstest. Beltz-Test. 2001.Google Scholar
- Hoppe C, Stojanovic J, Elger CE: Enhancing memory for lists by grouped presentation and rehearsal: a pilot study in healthy subjects with unexpected results. Seizure 2009, 18: 711-715. 10.1016/j.seizure.2009.10.001View ArticlePubMedGoogle Scholar
- LaViola JJ Jr: A discussion of cybersickness in virtual environments. ACM SIGCHI Bull 2000, 32: 47-56. 10.1145/333329.333344View ArticleGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.