- Open Access
A comparison of two personalization and adaptive cognitive rehabilitation approaches: a randomized controlled trial with chronic stroke patients
Journal of NeuroEngineering and Rehabilitation volume 17, Article number: 78 (2020)
Paper-and-pencil tasks are still widely used for cognitive rehabilitation despite the proliferation of new computer-based methods, like VR-based simulations of ADL’s. Studies have established construct validity of VR assessment tools with their paper-and-pencil version by demonstrating significant associations with their traditional construct-driven measures. However, VR rehabilitation intervention tools are mostly developed to include mechanisms such as personalization and adaptation, elements that are disregarded in their paper-and-pencil counterparts, which is a strong limitation of comparison studies. Here we compare the clinical impact of a personalized and adapted paper-and-pencil training and a content equivalent and more ecologically valid VR-based ADL’s simulation.
We have performed a trial with 36 stroke patients comparing Reh@City v2.0 (adaptive cognitive training through everyday tasks VR simulations) with Task Generator (TG: content equivalent and adaptive paper-and-pencil training). The intervention comprised 12 sessions, with a neuropsychological assessment pre, post-intervention and follow-up, having as primary outcomes: general cognitive functioning (assessed by the Montreal Cognitive Assessment - MoCA), attention, memory, executive functions and language specific domains.
A within-group analysis revealed that the Reh@City v2.0 improved general cognitive functioning, attention, visuospatial ability and executive functions. These improvements generalized to verbal memory, processing speed and self-perceived cognitive deficits specific assessments. TG only improved in orientation domain on the MoCA, and specific processing speed and verbal memory outcomes. However, at follow-up, processing speed and verbal memory improvements were maintained, and a new one was revealed in language. A between-groups analysis revealed Reh@City v2.0 superiority in general cognitive functioning, visuospatial ability, and executive functions on the MoCA.
The Reh@City v2.0 intervention with higher ecological validity revealed higher effectiveness with improvements in different cognitive domains and self-perceived cognitive deficits in everyday life, and the TG intervention retained fewer cognitive gains for longer.
The trial is registered at ClinicalTrials.gov, number NCT02857803. Registered 5 August 2016, .
Cognitive rehabilitation after stroke
Stroke is a leading cause of long-term acquired disability in adults , predisposing patients toward institutionalization and poorer quality of life . Over the coming decades, the incidence of post-stroke disability is expected to increase by 35% due to the rising prevalence of cerebrovascular risk and advances in medicine which are reducing post-stroke mortality rates . Historically, stroke rehabilitation has been focused on motor rehabilitation [4, 5]. However, post-stroke cognitive deficits are pervasive causing disability with major impacts on quality of life and independence on everyday life activities [6, 7]. In the last years, attention to the impact of cognitive deficits has been growing  and finding new ways to improve cognition after stroke is considered a priority . Also, more recently, the International Stroke Recovery and Rehabilitation Alliance 2018 working group has identified post-stroke cognitive impairments as a research priority .
Regardless of the many new developments in cognitive rehabilitation programs and applications, limited data on the effectiveness of cognitive rehabilitation is available because of the heterogeneity of participants, interventions, and outcome measures . Results from recent reviews corroborate that cognitive rehabilitation has a positive impact on post-stroke cognitive outcomes [12, 13], although of small magnitude (Hedges’ g = 0.48) . This result is in line with the quantitative  and qualitative [15,16,17] findings of previous reviews that have analyzed the effect of cognitive rehabilitation across multiple cognitive domains.
Is cognitive rehabilitation’s impact small or are we missing better cognitive rehabilitation methodologies?
Paper-and-pencil tasks are still the most widely used methods for cognitive rehabilitation because of their accessibility, ease of use, clinical validity and reduced cost . In the last years, computer-based versions of these traditional tasks are also starting to become clinically accepted [19, 20]. However, there is an absence of specific methodologies that inform health professionals which tasks to apply and under what clinical conditions . Consequently, rehabilitation professionals perform a selection of tasks based on their clinical experience, missing scientific foundations . We have proposed an objective and quantitative framework for the creation of personalized cognitive rehabilitation tasks based on a participatory design strategy with health professionals . In this work, through computational modeling, the authors operationalized 11 paper-and-pencil tasks and developed an Information and Communication Technologies based tool - the Task Generator (TG) - to tailor each of those 11 paper-and-pencil tasks to each patient in the domains of attention, memory, language and executive functions. A clinical evaluation of the TG with twenty stroke patients showed that the TG is able to adapt task parameters and difficulty levels according to patient’s cognitive assessment, and provide a comprehensive cognitive training . However, although it has been shown that rehabilitation strategies based on paper-and-pencil tasks can be personalized and adapted [24, 25], this approach presents a limited transfer to performance in activities of daily living (ADL) .
Over the last years, rehabilitation methodologies based on virtual reality (VR) have been developed as promising solutions to improve cognitive functions [26, 27]. VR-based tools have shown potential and to be ideal environments to incorporate cognitive tasks within the simulation of ADL’s . A recent trial with a VR-based simulation of everyday life activities (like going to the pharmacy, buying grocery at the supermarket, paying the water bill) suggested that an ecologically valid intervention has more impact than conventional methods (cognitive training using puzzles, calculus, problem resolution and shape sorting) in cognitive rehabilitation of stroke patients . Also, some of these VR-based systems allow the integration of motor training  and recent studies have already shown benefits of performing simultaneous motor and cognitive training with stroke patients using VR [31, 32]. Yet, there is still an insufficient number of rigorous trials to clinically validate VR methods  and there are difficulties associated with the limited access which results in a low adoption by health professionals who still prefer mostly use paper-and-pencil interventions .
In general, existing ecologically-valid VR-based environments are simulations of cities [29, 34,35,36,37,38], kitchens [39,40,41,42,43,44,45], streets [46,47,48,49,50,51], supermarkets [52,53,54,55,56], malls and other shopping scenarios [57,58,59,60,61]. Of these, only rare cases take into account training personalization according to patient cognitive profile and session-to-session adaptation [29, 36, 38, 41]. Additionally, the results of studies comparing VR cognitive interventions with standard occupational therapy or neuropsychology cognitive paper-and-pencil training are fundamentally subjective as control interventions. OT does not consider cognition as the main training focus, and neuropsychology paper-and-pencil training tasks are too similar to the cognitive assessment scales; additionally, both approaches do not incorporate personalization and dynamic adaptation to performance. Hence, even if rehabilitation sessions last the same, these interventions are not equivalent as they are delivered with uncontrolled difficulty levels and cognitive demands. Personalized rehabilitation is defined as involving an assessment of each patient’s impairments and performing a tailored intervention to his cognitive profile in the different domains. Instead, adaptation deals with the dynamic adjustment of the tasks’ cognitive demands according to the patients’ performance along the intervention sessions, therefore avoiding boredom (tasks that are to easy to solve) or frustration (tasks that are too difficult to solve).
Here we try to address some of the existing limitations in the validation of VR-based cognitive rehabilitation tools. In this study we compared two task content equivalent rehabilitation tools developed under the same personalization and adaptation framework : the TG and the Reh@City v2.0. This framework allows us to make sure that both tools deliver the same controlled adaptation and personalization of difficulty levels, and address the same cognitive demands. Hence, this comparison allows identifying the specific impact of increasing ecological validity of training through VR simulations of ADLs over the same training delivered through clinically accepted paper-and-pencil equivalent tasks. These findings will further inform on the specific benefits of ecologically valid environments delivered though VR and encourage the adoption of these technologies by health professionals.
Participants and trial design
Participants were selected based on the following inclusion criteria: no more than 75 years old; first stroke episode and at least at 6 months post-stroke (chronic phase); no hemi-spatial neglect as assessed by the clinicians with the Line Bisection test ; capacity to be seated; minimum of 2 years of schooling (since in Portugal there are quite high rates of illiteracy in the elder populations) and motivation to participate in the study. Patients with a total score of more than two standard deviations below the mean score for age and education in the Montreal Cognitive Assessment (MoCA) [63, 64] were excluded to ensure uniformity and enough cognitive capacity to participate in the rehabilitation interventions. Patients with severe depressive symptomatology, as assessed by the Beck Depression Inventory II [65, 66], were also excluded because its impact on cognitive functioning. Additionally, patients could not have been undergoing OT at least 2 months before the study. The study was previously approved by the Madeira Health Service Ethical Committee (reference number: 13/2016), and all the patients gave informed consent previous to participation.
The sample was selected from a list of 334 stroke patients enrolled in the cerebrovascular accidents appointment list from the Physical Medicine and Rehabilitation department of the Madeira Health Service (Portugal). They were contacted by phone by one of the researchers, and 44 declined to participate, 146 patients were excluded for not meeting the inclusion criteria and 108 were excluded for other reasons, such as transportation problems or lack of response after three phone calls. Overall, 36 patients were included meeting all inclusion criteria and were allocated to one of the two interventions (TG or Reh@City v2.0), by the two psychologists involved in the data collection. Group allocation was randomized (through a simple randomization method using a web-based application that generates a random allocation sequence) among the different rehabilitation units working under the Physical Medicine and Rehabilitation department (Fig. 1).
The study started in January 2017 and stopped in December 2018, since the authors defined the maximum of 2 years for data collection. In total, 19 participants were allocated to the TG group (one dropped out) and 17 allocated to the Reh@City v2.0 group (three dropped out). All patients went through neuropsychological assessment pre and post-intervention and at 2 months follow-up. Each one of the assessment times had an approximate duration of 90 min.
The intervention personalization was done through the characterization of each participant with the MoCA [63, 64] assessment results: the Attention parameter was defined from MoCA’s attention component score [0–6]; the delayed recall and orientation scores [0–11] was used to parameterize memory; executive functions was parameterized through the sum of the visuospatial, executive, and abstraction MoCA subscores [0–7]; MoCA’s naming and the language scores [0–6] was used to parameterize language; and the total score [0–30] was used to parameterize the overall difficulty.
Two psychologists performed all the assessments. The same two psychologists and one occupational therapist supervised the interventions sessions. Accordingly, this study was not blind.
Paper-and-pencil intervention: the task generator
The TG is a free and worldwide accessible tool that is able to generate personalized paper-and-pencil cognitive rehabilitation programs in PDF format, composed by a set of 11 tasks gathered from clinical settings and parameterized through a participatory process with rehabilitation experts : Cancellation; Numeric Sequences; Problem Resolution; Association; Comprehension of Contexts; Image Pairs; Word Search; Mazes; Categorization; Action Sequencing; and Memory of Stories (see Annex 1 for an example). After the characterization of each participant, the MoCA assessment data was normalized on a 1 to 10 scale, and a full training program was generated (Fig. 2). In the TG intervention, participants were instructed to use the arm they would feel more comfortable.
VR-based intervention: the Reh@City v2.0
Our VR-based intervention consisted of the same TG paper-and-pencil tasks contextualized in different locations of a virtual city with streets, sidewalks, buildings, shops, and parks – the Reh@City v2.0  (Table 1).
Reh@City v2.0 provides a more ecological training experience since patients are required to solve cognitive tasks through familiar ADL’s in a variety of commonplaces: for instance buy food in a supermarket (Fig. 3a); pick up a package in the post office; pay the electricity at the bank ATM (Fig. 3b); buy pain killers in the pharmacy; collect shirts in the clothing shop; play a game in the park (Fig. 3c); read the newspaper in the kiosk and set the table at home (Fig. 3d). These places display billboards and real products of actual spaces and trademarks commonly found in Portugal to help the patient relate the VR tasks to the real world. In addition, consistent with the actual simulated ADLs and to increase the ecological validity of the training, patients were also required to use their paretic arm to solve the tasks.
Because we were generally dealing with people of older age and low computer literacy, the interaction with the virtual environment was simplified and city was designed to have only square or rectangular building blocks and perpendicular street intersections, as well as simplified simulated environments. This simplified arrangement also allowed a more precise control of difficulty parameterization (Fig. 4).
VR simulations demand a trade-off of less realism to allow more real-time interactivity and difficulty parameterization. Notwithstanding this simplification, Reh@City has many elements attesting for its ecological validity. In terms of verisimilitude, the tasks simulate everyday life activities, although the performance of the virtual task is facilitated; the real environment and objects are minimally simulated and correspond to familiar locations and trademarks existing in Portugal; the user’s upper limb is partially represented; the specific situations in which actions take place are minimally modeled (for instance, action sequencing of baking a cake happens in a kitchen environment and reading a newspaper article happens in kiosk); it combines 2D, and 3D stimuli and; the navigation in the city is consistent with the point of view of the user. Concerning veridicality, in a previous study with Reh@City v1.0 , we have found a transfer of the virtual training to real-world functioning through significant improvements on a functional scale.
TG and Reh@City personalization and adaptation
The TG and the Reh@City v2.0 are two content equivalent rehabilitation tools developed under the same personalization algorithm :
where Ci indicates the contribution of each Independent Variable (IV) (task parameters) to the Dependent Variable (DV) (memory, executive functions, attention, language, and difficulty).
Concerning the session-to-session adaptation, when the patient finishes the first set of tasks, a score is computed using a 0 to 100% scale. Consistent with previous adaptive systems for stroke rehabilitation , if the mean performance is higher than 70%, the difficulty is increased by 0.5 in the next set of tasks, and if performance is from 0 to 50%, the difficulty parameter is reduced by 0.5.
In both personalization and adaptation, difficulty is one of the DV we have obtained from our algorithm , it is related with general cognitive demand.
Task generator (TG)
The TG is an online application, accessible at neurorehabilitation.m-iti.org/TaskGenerator, and does not require to be installed on the computer. Through this tool, clinicians defined appropriate parameters of training for memory, attention, executive functions, language, and difficulty (obtained from the MoCA screening tool), and it automatically generated a set of the 11 personalized cognitive training tasks. The only required software was a PDF reader to open the downloaded paper-and-pencil tasks. After printed, the tasks were solved on a table with a pencil having the user seated.
Reh@City v2.0 was installed on a PC (OS: Windows 7, CPU: Intel core 2 duo E8235 at 2.80 GHz, RAM: 4 Gb, Graphics: ATI mobility Radeon HD 2600 XT). Reh@City is a cognitive rehabilitation tool, that allow clinicians to personalize parameters of training for memory, attention, executive functions, language, and difficulty according to the MoCA total and subdomains score (Fig. 5) and it enables that through upper limb movements.
Given the potential benefits reported in the literature of combining cognitive and motor rehabilitation through VR [30,31,32], Reh@City v2.0 implies the use of the paretic arm to solve its cognitive training tasks. The user worked on a tabletop, facing an LCD monitor (24″) and moved a customized handle with a tracking pattern on the surface of the table with his/her paretic arm (Fig. 6). 2D upper limb reaching movements were captured through a camera-based Augmented Reality (AR) pattern tracking software (AnTS)  connected to a PlayStation Eye camera (Sony Computer Entertainment Inc., Tokyo, Japan). For adapting the interaction to individual users, the Reh@City v2.0 implemented a built-in calibration function that normalizes the motor effort required in the task to the active range of movement of the user. The movements of the user are then mapped onto the movements of a virtual arm (in indoor tasks) or as movement directions (during outdoor navigation) in the Reh@City v2.0 environment.
Primary outcome measures: general cognitive functioning, attention, memory, executive functions and language
As primary outcome measures we used the MoCA [63, 64] as a general cognitive functioning measure, which has been reported to have a good sensitivity and specificity in screening for cognitive impairment after stroke . In addition, the decline in MoCA scores (reduction ≥2 points) was found to be associated with the decline in neuropsychological diagnosis transitional status on a sample of 275 stroke patients .
Moreover, we selected specific attention, memory, executive functions, and language assessments, which are the domains targeted by both TG and Reh@City v2.0. To assess attention we used the Trail Making Test A and B (TMT A and B) [72, 73], a very popular neuropsychological test that provides information on visual search, visual scanning, selective and divided attention, processing speed, mental flexibility, and also executive functioning. In part A, circles numbered from 1 to 25 need to be connected in numerical order. In part B, numbers from 1 to 13 and letters from A to L need to be connected alternating numbers and letters in ascending order. The memory assessment was performed with the Verbal Paired Associates from the Wechsler Memory Scale-III (WMS-III) . To assess executive functions, namely working memory and processing speed, we used the Digit Span (forward and backward recall conditions) also from the WMS-III, and the Symbol Search and the Digit Symbol Coding (codification and incidental learning pairing conditions) from the Wechsler Adult Intelligence Scale III (WAIS) . Finally, we assessed language through the Vocabulary from WAIS-III, which provides information about verbal comprehension.
Secondary outcome measure: self-perceived impact of cognitive functioning problems
As secondary outcome measure, we assessed the perceived impact of persisting problems with cognition, as assessed by the Patient-Reported Evaluation of Cognitive State (PRECiS) [76, 77], which includes 27 core items asking respondents about the impact of cognition on four conceptual dimensions: everyday life skills, family and life, mood and sense of self.
All statistical analyses were performed using SPSS software (version 20, SPSS Inc., Chicago IL, USA). As a criterion for significance, we used a α of .050. With Bonferroni correction the α was .002, as such significant differences with corrected p-values are also mentioned. Normality of data was assessed with the Kolmogorov-Smirnov (KS) test. As some data were not normally distributed, nonparametric tests were used to evaluate the inter-group and intra-group differences. The Wilcoxon signed-rank test (W) was used to analyze the within group changes over time, while the two-tailed Mann-Whitney (MW) test was used to compare the between-group differences from baseline to the end of the study. Demographic differences between groups were measured with the Mann Whitney test (MW). Effect sizes (r) were computed as Z/√N on the pairwise comparisons. The criteria for interpretation of the effect was 0.1 = small, 0.3 = medium, and 0.5 = large.
The sample consisted of thirty-two patients with stroke randomly distributed in two groups. The Reh@City v2.0 group comprised fourteen (5 male, 9 female) adult (M = 59.1 years old, SD = 11.8) patients with stroke (11 right hemisphere, 3 left hemisphere; 12 ischemic, 2 hemorrhagic), with an average of 45.9 ± 43.6 months post- stroke and a mean of 8 ± 5.3 years of schooling. The TG group comprised eighteen (11 male, 7 female) adult (M = 65 years old, SD = 6.2) patients with stroke (9 right hemisphere, 6 left hemisphere, 3 not specified; 14 ischemic, 3 hemorrhagic, 1 not specified), with an average of 21.3 ± 12.9 months post-stroke and a mean of 5.5 ± 3.2 years of schooling. The Mann-Whitney test revealed no differences between groups in the demographic characteristics and in all baseline outcome measures (Table 2).
According to the Kolmogorov-Smirnov (KS) test, data were normally distributed in both groups for age (KSReh@City = .189, p = .190; KSTG = .182, p = .118) and time post-stroke (KSReh@City = .211, p = .091; KSTG = .187, p = .095). Data were not normally distributed for gender (KSReh@City = .407, p = <.001; KSTG = .392, p = <.001), type of stroke (KSReh@City = .510, p = <.001; KSTG = .463, p = <.001), the side of lesion (KSReh@City = .510, p = <.001; KSTG = .463, p = <.001), and the number of years of schooling (KSReh@City = .345, p = < .001; KSTG = .405, p = < .001).
Primary outcome measures
MoCA - general cognitive functioning
We analyzed the global cognitive functioning, as assessed by the MoCA, of the two groups in the pre and post intervention assessments, and follow-up for the Reh@City v2.0 and TG groups (Table 3).
A Wilcoxon test for within-groups differences revealed that only the Reh@City v2.0 group presented significant statistical improvements between pre and post assessment times in MoCA [Pre: Mdn = 23, IQR = 19.8–26; Post: Mdn = 25, IQR = 23–27.3 (W(14) = 64.00, Z = − 2.777, p = .005, r = .74)]. In the subdomains analysis, we found significant improvements in visuospatial ability and executive functioning [Pre: Mdn = 3.5, IQR = 2.8–4; Post: Mdn = 4, IQR = 3–5 (W(14) = 41.00, Z = − 2.310, p = .021, r = .62)] and attention [Pre: Mdn = 4, IQR = 2.8–5.3; Post: Mdn = 5.5, IQR = 3–6 (W(14) = 28.00, Z = − 2.460, p = .014, r = .66)].
Concerning the TG group, the only significant change was in the MoCA orientation subdomain [Pre: Mdn = 6, IQR = 5–6; Post: Mdn = 6, IQR = 6–6 (W(18) = 15.00, Z = − 2.121, p = .034, r = .57)].
A Mann-Whitney test indicated that the Reh@City v2.0 group improved significantly more than the TG group, in terms of general cognitive functioning, as assessed by the MoCA, from baseline to post-intervention [Reh@City v2.0: Mdn = 2, IQR = 0–3; TG: Mdn = − 1.5, IQR = − 3.25–2 (U = 65.00, Z = − 2.334, p = .020, r = .41)].
TMT A & B - attention
We computed the TMT A and TMT B performance for the two groups, in terms of errors and completion time, pre, post-intervention and follow-up (Table 4). Only the TG group showed a significant improvement in the reduction of time to completion of the TMT A test pre to post-intervention [Pre: Mdn = 84, IQR = 59.5–114.3; Post: Mdn = 72, IQR = 58.8–99.5 (W(18) = 18.00, Z = − 2.588, p = .010, r = .61)].
WMS-III verbal paired associates - memory
Table 4 describes the Verbal Paired Associates test performance for the two groups pre, post-intervention and follow-up. In this learning and memory test, we found significant improvements within the Reh@City v2.0 group for the retention [Pre: Mdn = 75, IQR = 0–100; Post: Mdn = 100, IQR = 74.1–100 (W(14) = 36.00, Z = − 2.524, p = .012, r = .67)] and recognition [Pre: Mdn = 24, IQR = 21.8–24; Post: Mdn = 24, IQR = 24–24 (W(14) = 21.00, Z = − 2.214, p = .027, r = 59)] scores post-intervention. In the TG group improvements were only significant for the retention score in both post-intervention [Pre: Mdn = 0, IQR = 0–56.3; Post: Mdn = 82.9, IQR = 26.5–100 (W(18) = 118.00, Z = − 2.602, p = .009, r = .61)] and follow-up [Pre: Mdn = 0, IQR = 0–56.3; FU: Mdn = 82.9, IQR = 37.5–100 (W(18) = 95.00, Z = − 2.776, p = .006, r = .65)].
WAIS-III Digit Symbol Coding, Symbol Search and Digit Span - Executive Functions.
Table 4 describes the executive functioning outcome measures for the two groups pre, post-intervention and follow-up. The Reh@City v2.0 group showed improvements in the Digit Symbol Coding codification task post-intervention [Pre: Mdn = 28.5, IQR = 23.5–36.8; Post: Mdn = 33, IQR = 26.8–47 (W(14) = 87.00, Z = − 2.171, p = .030, r = .58)]. The TG group had significant improvements in the Symbol Search at follow-up [Pre: Mdn = 12, IQR = 7.8–13.5; FU: Mdn = 15, IQR = 9–20.3 (W(18) = 101.00, Z = − 2.340, p = .019, r = .55)].
WAIS-III vocabulary - language
The analysis of the language outcome measure for the two groups pre, post-intervention and follow-up revealed that only the TG group showed improvements in the Vocabulary assessment at follow-up [Pre: Mdn = 19.5, IQR = 13–28.5; FU: Mdn = 24.5, IQR = 16.5–30.3 (W(18) = 166.00, Z = − 3.514, p < .001, r = .83)] (Table 4). This difference was also significant with Bonferroni correction (p = .002).
Secondary outcome measure
Patient-reported evaluation of cognitive state
When analyzing the answers of the two groups pre, post-intervention and follow-up to the PRECiS questionnaire, only the Reh@City v2.0 group revealed a significant self-perceived decrease in the stroke cognitive deficits impact post intervention [Pre: Mdn = 13.5, IQR = 7–23.8; Post: Mdn = 12, IQR = 3.8–21.3 (W(14) = 13.00, Z = −2.041, p = .041, r = .55)] (Table 5).
In the last years there has been significant growth in the evidence for post-stroke cognitive rehabilitation [12, 13], with a number of studies proposing ecologically valid VR-based simulations of ADL’s as the most promising training solutions [26, 27]. Notwithstanding the advantages of VR, there are several areas that require further progress. One is the need to bridge widely accepted paper-and-pencil methodologies with VR-based ADL’s simulations. In the field of cognitive assessment, some studies have compared VR cognitive assessment tools with their paper-and-pencil original versions [78,79,80]. Existing studies about VR simulations of cities showed convergent validity between some measures of the performance in simulated virtual cities and clinical neuropsychological tests of variable strength, which ranges from moderate for attention , good to excellent for executive functioning , and excellent for general cognitive condition . Poor to moderate correlations have also been reported between navigation in a real and a virtual city . If we look at cognitive training, to the best of our knowledge, there is not much work that we can compare with our study. In terms of VR-based interventions, Gamito and colleagues used a virtual city with everyday life simulations with stroke patients and have found improvements in attention and memory . Related with Task Generator paper-and-pencil training, there are the Guttman Neuropersonal Trainer  and the Cogweb  that are customizable and have already some evidence about its efficacy in cognitive training.
Another limitation about VR-based cognitive rehabilitation tools is that, an important number of the developed systems have not been field tested , have only gone through studies with a small number of participants [29, 36] and/or with healthy control groups . Additionally, none of these VR-based ADL’s simulations are compared with widely used and clinically accepted paper-and-pencil tasks, being always compared with non-equivalent interventions as OT . Our RCT, is the first to implement an adaptive paper-and-pencil training and compare it with a content equivalent VR-based ADL’s simulation, by using the same tasks, personalization and difficulty adaptation framework within a longitudinal clinical intervention.
TG and the Reh@City v2.0 groups differed in the motor demand of the training; while the TG group used a pencil with the healthy arm to solve the tasks, the Reh@City v2.0 group used an adapted handle with the affected arm, performing 2D upper limb reaching movements to interact with the tasks. Assuming the interdependence between the recovery processes , we may have provided a more effective rehabilitation to the Reh@City v2.0 group in both cognitive and motor domains; however, motor recovery is not an outcome in this study and, consequently, was not assessed. The Reh@City v2.0 main goal is to be an ecologically valid cognitive training tool, and the use and virtual representation of the paretic arm to solve the ADL’s simulations was implemented to increase ecological validity. As such, the difference in motor demand is intrinsic to the research hypothesis. If we asked the TG group to use the paretic arm, we would be harming the resolution of the paper-and-pencil tasks, thereby, Reh@City v 2.0 and TG interventions could not be fairly compared.
Primary outcome measures
The Reh@City v2.0 group improved in the MoCA general cognitive functioning and in its attention, visuospatial ability and executive functioning subdomains, with large effect sizes. The TG group improved in the MoCA orientation domain, also with a large effect size. In a between groups comparison, the Reh@City v2.0 group had a higher impact in the general cognitive functioning comparatively to the TG group, with a medium effect size. Wong and colleagues (2017) determined that the MoCA score change associated with a change of health in general, in a sample of 175 aneurysmal subarachnoid hemorrhage patients, was of two points . The generalization of this finding to our study is consistent with this between-groups difference, since the Reh@City v2.0 group improves its median score from 23 to 25 points in the post-intervention, while the TG maintains the median score of 21 points. At the follow-up, Reh@City v2.0 improves its median score to 28 points, while the TG has a more discrete improvement to 23 points. Meaning that the Reh@City v2.0 group was the only to present Minimum Clinically Important Differences (MCID) from baseline to end, and both from end to follow-up.
Despite the fact that both Reh@City v2.0 and TG are content equivalent interventions that follow the same personalization guidelines and difficulty progression rules, the superiority of the impact of the VR-based intervention was expected since it reunites a number of promising features that distinguishes it from the paper-and-pencil intervention: 1) it is an ecologically valid ADL’s simulations; 2) stimulus are everyday-life brands and products; 3) it has game elements so participants are rewarded for successful performance in real time and; 4) it provides immediate feedback enabling higher success in accomplishing the tasks. Although with a different primary outcome measure, these results are coherent with a previous Reh@City v1.0 study, where the Reh@City v1.0 group improved in the Addenbrooke Cognitive Examination  general cognitive functioning, attention, memory and visuospatial abilities and was superior between groups in general cognitive functioning, attention and fluency (where the control group had a significant decrease) . Reh@City v1.0 was an initial prototype with four commonly frequented places of daily life (Pharmacy, Bank, Supermarket, and Post-office) . The upgrade to Reh@City v2.0 included: an increase of the ecological validity through the improvement in the overall visual realism of the city and existing tasks, and the interaction through the paretic arm using an adapted handle; the implementation of dynamic difficulty adaptation based on the framework we developed for TG tasks ; and the increase of the number of cognitive training tasks and locations (Magazine Kiosk, Home, Park, and Fashion Store) .
Also, the Reh@City v2.0 intervention group had a significant impact, and with large effect sizes, in verbal memory (as assessed by the retention and recognition from the Verbal Paired Associates test - WMS-III), and processing speed (as assessed by the Digit Symbol Coding task - WAIS-III), which is superior to what we have found in the Reh@City v1.0 study, where we only had improvements in the executive functioning measure (the Picture Arrangement test from the WAIS-III) and the control group improved in the reduction of the number of errors in the TMT B, a processing speed and selective and divided attention measure . This superiority of the Reh@City v2.0 concerning verbal memory and processing speed was expected since it has an increased number of cognitive training tasks and locations in comparison to Reh@City v1.0, that did not even had verbal memory training.
The TG group improved in verbal memory (as assessed by the retention from the Verbal Paired Associates test - WMS-III) and processing speed (as assessed by the TMT A task execution time) subdomains, with large effect sizes. At follow-up, participants who underwent the TG intervention maintained the verbal memory benefits with new improvements in the sustained attention and processing speed (as assessed by the Symbol Search task – WAIS-III) and language (as assessed by the Vocabulary – WAIS-III) domains, also with large effect sizes. These findings may be related to the fact that the TG offers a more domain-specific training and recent evidence supports that attention, language , memory, executive functions and visuospatial and perceptual skills  training after stroke is effective.
Secondary outcome measure
Besides cognition, we assessed the intervention’s impact in the self-perceived cognitive deficits. Only the Reh@City v2.0 group had a significant reduction in the self-perceived cognitive deficits in different aspects of their everyday life (everyday life skills, family and life, mood and sense of self), measured by the PRECiS questionnaire with a large effect size. This finding is in line with existing literature that states that, comprehensive neuropsychological rehabilitation, is effective to reduce functional disability after stroke . In the Reh@City v1.0 study we assessed the intervention’s impact in the multiple domains of health and life with a different outcome measure - the Stroke Impact Scale 3.0 . In this self-report questionnaire, the VR group improved significantly in the physical domain, memory, emotion, social participation and overall recovery while the control group decreased in the physical domain, only improving in memory, mobility and social participation, which supported the superiority of the intervention with the Reh@City v1.0 in comparison to the control group.
Some limitations of our study must be considered when interpreting the results. Concerning the sample, although 42 stroke participants is a small sample it is comparable to previous similar clinical trials [29, 36]. In the perspective of the interventions comparison, Reh@City v2.0 and TG have several intrinsic differences mainly related with the fact that cognitive tasks are presented differently (through VR and paper-and-pencil, respectively). Besides the novelty carried by the use of VR technology in Reh@City v2.0, it requires the subject to use its paretic upper limb while TG does not, as it might be hard or even impossible for a participant to properly use a pencil to solve cognitive tasks with his paretic arm. Although the discrepancies inherent to both interventions do not allow us to know if Reh@City v2.0 impact superiority is related to the use of VR, to the ecological validity of the tasks, or the integration of motor and cognitive training, the main objective of this study was to compare Reh@City v2.0 with a content equivalent intervention that follows the same personalization guidelines and difficulty progression rules (the TG), acknowledging their differences. Also, in the Reh@City v2.0 group, most participants were lost at follow-up. Hence, the comparison between TG and Reh@City v2.0 at this assessment moment should be considered with caution. Furthermore, the intervention was not blind since the same persons performed the assessments and interventions. Also, there might have been learning effects of the cognitive assessment tools, at post-intervention and follow-up assessment times, since only the MoCA had parallel versions for multiple assessments. Yet, even if a learning effect existed, this would apply to both groups, and the comparison would still be valid.
The results of this one-month longitudinal study showed a positive impact of a rehabilitation training with the Reh@City v2.0, an ecologically valid VR ADL’s simulations, in general cognitive functioning, visuospatial ability and executive functioning, attention, verbal memory and processing speed, generalized for other health and life aspects measured by the self-perceived impact of cognitive deficits scale. This generalization did not happen in the TG group that only revealed similar cognitive impact in the orientation, processing speed and verbal memory domains. The TG intervention sustained impact at follow-up, maintaining processing speed and verbal memory improvements and revealing a new one in language. Finally, by comparing interventions between themselves, we have found Reh@City v2.0 to be superior in general cognitive functioning, visuospatial ability and executive functioning. Only the intervention with TG allowed cognitive gains to last over time. However, these results need to be considered with caution given the dropout at follow-up in the Reh@City v2.0 group.
Overall, our results contribute with new evidence about the impact of ecological validity - using personalization and adaptation in VR simulations of ADL’s and paper-and-pencil tasks - in the rehabilitation of cognitive deficits, which can facilitate the adoption of these innovative tools by health professionals in their daily practice. Nevertheless, there is still a need for further research considering other clinical populations, as well as the implementation of a wider variety of cognitive training tasks.
Availability of data and materials
The central tendency and dispersion data, as well as the Mann-Whitney and the Wilcoxon tests, from which the conclusions are drawn, are provided in the article. Raw data is available from the corresponding author on reasonable request.
Activities of Daily Living
Montreal Cognitive Assessment
Patient-Reported Evaluation of Cognitive State
Randomized Controlled Trial
Trail Making Test
Wechsler Adult Intelligence Scale
Wechsler Memory Scale
Feigin VL, Norrving B, Mensah GA. Global burden of stroke. Circ Res. 2017;120(3):439–48.
Ankolekar S, Renton C, Sare G, Ellender S, Sprigg N, Wardlaw JM, et al. Relationship between poststroke cognition, baseline factors, and functional outcome: data from “efficacy of nitric oxide in stroke” trial. J Stroke Cerebrovasc Dis. 2014;23(7):1821–9.
Bennett JE, Stevens GA, Mathers CD, Bonita R, Rehm J, Kruk ME, et al. NCD countdown 2030: worldwide trends in non-communicable disease mortality and progress towards sustainable development goal target 3.4. Lancet. 2018;392(10152):1072–88.
Dimyan MA, Cohen LG. Neuroplasticity in the context of motor rehabilitation after stroke. Nat Rev Neurol. 2011;7(2):76–85.
Ganguly K, Byl NN, Abrams GM. Neurorehabilitation: motor recovery after stroke as an example. Ann Neurol. 2013;74(3):373–81.
Cumming TB, Marshall RS, Lazar RM. Stroke, cognitive deficits, and rehabilitation: still an incomplete picture. Int J Stroke. 2012;8(1):38–45.
Tang EY, Amiesimaka O, Harrison SL, Green E, Price C, Robinson L, et al. Longitudinal effect of stroke on cognition: a systematic review. J Am Heart Assoc. 2018;7(2):e006443.
Andrew NE, Kilkenny M, Naylor R, Purvis T, Lalor E, Moloczij N, et al. Understanding long-term unmet needs in Australian survivors of stroke. Int J Stroke. 2014;9(SA100):106–12.
Pollock A, St George B, Fenton M, Firkins L. Top 10 research priorities relating to life after stroke – consensus from stroke survivors, caregivers, and health professionals. Int J Stroke. 2014;9(3):313–20.
Bernhardt J, Borschmann KN, Kwakkel G, Burridge JH, Eng JJ, Walker MF, et al. Setting the scene for the second stroke recovery and rehabilitation roundtable. Int J Stroke. 2019. https://0-doi-org.brum.beds.ac.uk/10.1177/1747493019851287.
Sohlberg MM, Mateer CA. Cognitive rehabilitation: an integrative neuropsychological approach. New York: Guilford; 2001.
Aminov A, Rogers JM, Middleton S, Caeyenberghs K, Wilson PH. What do randomized controlled trials say about virtual rehabilitation in stroke? A systematic literature review and meta-analysis of upper-limb and cognitive outcomes. J Neuroeng Rehabil. 2018;15(1):29.
Rogers JM, Foord R, Stolwyk RJ, Wong D, Wilson PH. General and domain-specific effectiveness of cognitive remediation after stroke: systematic literature review and Meta-analysis. Neuropsychol Rev. 2018;28(3):285–309.
Cicerone K, Goldin Y, Ganci K, Rosenbaum A, Wethe J, Langenbahn D, et al. Evidence-based cognitive rehabilitation: systematic review of the literature from 2009 through 2014. Arch Phys Med Rehabil. 2019;S0003-9993(19):30194–7.
Cicerone KD, Dahlberg C, Kalmar K, Langenbahn DM, Malec JF, Bergquist TF, et al. Evidence-based cognitive rehabilitation: recommendations for clinical practice. Arch Phys Med Rehabil. 2000;81(12):1596–615.
Cicerone KD, Dahlberg C, Malec JF, Langenbahn DM, Felicetti T, Kneipp S, et al. Evidence-based cognitive rehabilitation: updated review of the literature from 1998 through 2002. Arch Phys Med Rehabil. 2005;86(8):1681–92.
van Heugten C, Gregório GW, Wade D. Evidence-based cognitive rehabilitation after acquired brain injury: a systematic review of content of treatment. Neuropsychol Rehabil. 2012;22(5):653–73.
Parsons TD. Neuropsychological Rehabilitation 3.0: State of the Science. In: Clinical Neuropsychology and Technology: Springer International Publishing; 2016. p. 113–32. Available from: http://0-link.springer.com.brum.beds.ac.uk/chapter/10.1007/978-3-319-31075-6_7. Cited 2016 Apr 8.
Solana J, Cáceres C, García-Molina A, Opisso E, Roig T, Tormos JM, et al. Improving brain injury cognitive rehabilitation by personalized Telerehabilitation services: Guttmann Neuropersonal trainer. IEEE J Biomed Health Inform. 2015;19(1):124–31.
Tedim Cruz V, Pais J, Ruano L, Mateus C, Colunas M, Alves I, et al. Implementation and Outcomes of a Collaborative Multi-Center Network Aimed at Web-Based Cognitive Training – COGWEB Network. JMIR Ment Health. 2014;1(1) Available from: http://0-www.ncbi.nlm.nih.gov.brum.beds.ac.uk/pmc/articles/PMC4607396/. Cited 2016 May 2.
Gracey F, Wilson BA. Theoretical approaches to cognitive rehabilitation. In: Goldstein LH, McNeil JE, editors. Clinical Neuropsychology: A Practical Guide to Assessment and Management for Clinicians: Wiley; 2013. p. 463–6.
Wood RL. Towards a model of cognitive rehabilitation. In: Wood RL, Fussey I, editors. Cognitive rehabilitation in perspective. Routledge; 2018. p. 3–26.
Faria AL, Pinho MS, Bermúdez i Badia S. Capturing expert knowledge for the personalization of cognitive rehabilitation: study combining computational modeling and a participatory design strategy. JMIR Rehabil Assist Technol. 2018;5(2):e10714.
Faria AL & Bermúdez i Badia S. Personalizing paper-and-pencil training for cognitive rehabilitation. Funchal: International Conference on Applied Psychology and Human Behavior proceedings; 2018. p. 1–6.
Faria A, Bermúdez i Badia S. Development and evaluation of a web-based cognitive task generator for personalized cognitive training: a proof of concept study with stroke patients. In: REHAB 2015: 3rd Workshop on ICTs for improving patients research techniques. Lisboa: ACM; 2015.
Luca RD, Russo M, Naro A, Tomasello P, Leonardi S, Santamaria F, et al. Effects of virtual reality-based training with BTs-nirvana on functional recovery in stroke patients: preliminary considerations. Int J Neurosci. 2018;128(9):791–6.
Maggio MG, Latella D, Maresca G, Sciarrone F, Manuli A, Naro A, et al. Virtual reality and cognitive rehabilitation in people with stroke: an overview. J Neurosci Nurs. 2019;51(2):101–5.
Vourvopoulos A, Faria AL, Ponnam K, Bermudez i Badia S. RehabCity: Design and Validation of a Cognitive Assessment and Rehabilitation Tool Through Gamified Simulations of Activities of Daily Living. In: Proceedings of the 11th Conference on Advances in Computer Entertainment Technology. New York: ACM; 2014. 26:1–26:8. https://0-doi-org.brum.beds.ac.uk/10.1145/2663806.2663852. (ACE ’14). Cited 2016 Dec 2.
Faria AL, Andrade A, Soares L, Bermúdez i Badia S. Benefits of virtual reality based cognitive rehabilitation through simulated activities of daily living: a randomized controlled trial with stroke patients. J Neuroeng Rehabil. 2016;13(1):96.
Hagovska M, Nagyova I. The transfer of skills from cognitive and physical training to activities of daily living: a randomised controlled study. Eur J Ageing. 2016;14(2):133–42 Levin MF, Weiss PL, Keshner EA. Emergence of virtual reality as a tool for upper limb rehabilitation: incorporation of motor control and motor learning principles. Phys Ther 2015; 95(3):415–425.
Kizony R, Katz N, Weiss PL. Virtual reality based intervention in rehabilitation: relationship between motor and cognitive abilities and performance within virtual environments for patients with stroke. In: Proceedings of the 5th international conference on disability, virtual reality and associated technology Oxford, UK; 2004. Kizony R, Levin MF, Hughey L, Perez C, Fung J. Cognitive load and dual-task performance during locomotion poststroke: a feasibility study using a functional virtual environment. Phys Ther 2010;90(2):252–60.
Subramanian SK, Chilingaryan G, Levin MF, Sveistrup H. Influence of training environment and cognitive deficits on use of feedback for motor learning in chronic stroke. In: 2015 International Conference on Virtual Rehabilitation (ICVR): IEEE; 2015. p. 38–43. Mullick AA, Subramanian SK, Levin MF. Emerging evidence of the association between cognitive deficits and arm motor recovery after stroke: a meta-analysis. Restor Neurol Neurosci 2015;33(3):389–403.
WHO. WHO | Assistive devices/technologies: what WHO is doing: WHO; 2011. Available from: http://www.who.int/disabilities/technology/activities/en/. Cited 2016 May 9.
Claessen MH, Visser-Meily JM, Rooij NK, Postma A, Ham IJ. A direct comparison of real-world and virtual navigation performance in chronic stroke patients. J Int Neuropsychol Soc. 2016;22(4):467–77.
Dores AR, Miranda MJ, Carvalho IP, Mendes L, Barbosa F, Coelho A, et al. Virtual City: Neurocognitive rehabilitation of Acquired Brain Injury. In: 2012 7th Iberian Conference on Information Systems and Technologies (CISTI); 2012. p. 1–4.
Gamito P, Oliveira J, Coelho C, Morais D, Lopes P, Pacheco J, et al. Cognitive training on stroke patients via virtual reality-based serious games. Disabil Rehabil. 2015;39(4):385–8.
Jovanovski, Zakzanis K, Ruttan L, Campbell Z, Erb S, Nussbaum D. Ecologically valid assessment of executive dysfunction using a novel virtual reality task in patients with acquired brain injury. Applied Neuropsychol Adult. 2012;19(3):207–20.
Klinger E, Joseph P-A, Le Guiet J-L, Fuchs P, du Lac N, Servant F. AGATHE: A Tool for Personalized Rehabilitation of Cognitive Functions. In: ACM SIGGRAPH 2013 Emerging Technologies. New York: ACM; 2013. p. 1:1. (SIGGRAPH ‘13). Available from: http://0-doi.acm.org.brum.beds.ac.uk/10.1145/2503368.2503369. Cited 2016 Dec 2.
Adams RJ, Lichter MD, Krepkovich ET, Ellington A, White M, Diamond PT. Assessing upper extremity motor function in practice of virtual activities of daily living. IEEE Trans Neural Syst Rehabil Eng. 2015;23(2):287–96.
Besnard J, Richard P, Banville F, Nolin P, Aubin G, Le Gall D, et al. Virtual reality and neuropsychological assessment: the reliability of a virtual kitchen to assess daily-life activities in victims of traumatic brain injury. Appl Neuropsychol Adult. 2015;23(3):223–35.
Cao X, Douguet A-S, Fuchs P, Klinger E. Designing an ecological virtual task in the context of executive functions: Preliminary study; 2010. p. 71–7. Available from: https://hal-mines-paristech.archives-ouvertes.fr/hal-00785344. Cited 2016 Dec 13.
Christiansen C, Abreu B, Ottenbacher K, Huffman K, Masel B, Culpepper R. Task performance in virtual environments used for cognitive rehabilitation after traumatic brain injury. Arch Phys Med Rehabil. 1998;79(8):888–92.
Edmans JA, Gladman JRF, Cobb S, Sunderland A, Pridmore T, Hilton D, et al. Validity of a virtual environment for stroke rehabilitation. Stroke. 2006;37(11):2770–5.
Zhang L, Abreu BC, Seale GS, Masel B, Christiansen CH, Ottenbacher KJ. A virtual reality environment for evaluation of a daily living skill in brain injury rehabilitation: reliability and validity. Arch Phys Med Rehabil. 2003;84(8):1118–24.
O’Brien J. Simulating the homes of stroke patients: can virtual environments help to promote engagement in therapy activities? In: 2007 Virtual Rehabilitation; 2007. p. 23–8.
Devos H, Akinwuntan AE, Nieuwboer A, Tant M, Truijen S, De Wit L, et al. Comparison of the effect of two driving retraining programs on on-road performance after stroke. Neurorehabil Neural Repair. 2009;23(7):699–705.
Sorita E, N’Kaoua B, Larrue F, Criquillon J, Simion A, Sauzéon H, et al. Do patients with traumatic brain injury learn a route in the same way in real and virtual environments? Disabil Rehabil. 2013;35(16):1371–9.
Katz N, Ring H, Naveh Y, Kizony R, Feintuch U, Weiss PL. Interactive virtual environment training for safe street crossing of right hemisphere stroke patients with unilateral spatial neglect. Disabil Rehabil. 2005;27(20):1235–44.
Lloyd J, Riley GA, Powell TE. Errorless learning of novel routes through a virtual town in people with acquired brain injury. Neuropsychol Rehabil. 2009;19(1):98–109.
Navarro M-D, Lloréns R, Noé E, Ferri J, Alcañiz M. Validation of a low-cost virtual reality system for training street-crossing. A comparative study in healthy, neglected and non-neglected stroke individuals. Neuropsychol Rehabil. 2013;23(4):597–618.
Titov N, Knight RG. A computer-based procedure for assessing functional cognitive skills in patients with neurological injuries: the virtual street. Brain Inj. 2005;19(5):315–22.
Josman N, Kizony R, Hof E, Goldenberg K, Weiss PL, Klinger E. Using the virtual action planning-supermarket for evaluating executive functions in people with stroke. J Stroke Cerebrovasc Dis. 2014;23(5):879–87.
Kang YJ, Ku J, Han K, Kim SI, Yu TW, Lee JH, et al. Development and clinical trial of virtual reality-based cognitive assessment in people with stroke: preliminary study. CyberPsychol Behav. 2008;11(3):329–39.
Raspelli S, Pallavicini F, Carelli L, Morganti F, Pedroli E, Cipresso P, et al. Validating the Neuro VR-based virtual version of the multiple errands test: preliminary results. Presence Teleop Virt. 2012;21(1):31–42.
Sorita E, Joseph PA, N’kaoua B, Ruiz J, Simion A, Mazaux JM, et al. Performance analysis of adults with acquired brain injury making errands in a virtual supermarket. Ann Phys Rehabil Med. 2014;57:e85.
Yip BC, Man DW. Virtual reality-based prospective memory training program for people with acquired brain injury. Neurorehabilitation. 2013;32(1):103–15.
Canty AL, Fleming J, Patterson F, Green HJ, Man D, Shum DHK. Evaluation of a virtual reality prospective memory task for use with individuals with severe traumatic brain injury. Neuropsychol Rehabil. 2014;24(2):238–65.
Erez N, Weiss PL, Kizony R, Rand D. Comparing performance within a virtual supermarket of children with traumatic brain injury to typically developing children: a pilot study. OTJR Occup Participation Health. 2013;33(4):218–27.
Jacoby M, Averbuch S, Sacher Y, Katz N, Weiss PL, Kizony R. Effectiveness of executive functions training within a virtual supermarket for adults with traumatic brain injury: a pilot study. IEEE Trans Neural Syst Rehabil Eng. 2013;21(2):182–90.
Okahashi S, Seki K, Nagano A, Luo Z, Kojima M, Futaki T. A virtual shopping test for realistic assessment of cognitive function. J Neuroeng Rehabil. 2013;10:59.
Rand D, Rukan SB-A, Weiss PL, Katz N. Validation of the virtual MET as an assessment tool for executive functions. Neuropsychol Rehabil. 2009;19(4):583–602.
Schenkenberg T, Bradford DC, Ajax ET. Line bisection and unilateral visual neglect in patients with neurologic impairment. Neurology. 1980;30(5):509.
Nasreddine ZS, Collin I, Chertkow H, Phillips N, Bergman H, Whitehead V. Sensitivity and specificity of the Montreal cognitive assessment (MoCA) for detection of mild cognitive deficits. Can J Neurol Sci. 2003;30(2):30.
Freitas S, Simões MR, Alves L, Santana I. Montreal cognitive assessment (MoCA): normative study for the Portuguese population. J Clin Exp Neuropsychol. 2011;33(9):989–96.
Beck AT, Ward CH, Mendelson M, Mock J, Erbaugh J. An inventory for measuring depression. Arch Gen Psychiatry. 1961;4:561–71.
Campos RC, Gonçalves B. The Portuguese version of the Beck depression inventory-II (BDI-II): preliminary psychometric data with two nonclinical samples. Eur J Psychol Assess. 2011;27(4):258–64.
Paulino T, Faria AL, Bermúdez i Badia S. Reh@ City v2. 0: a comprehensive virtual reality cognitive training system based on personalized and adaptive simulations of activities of daily living. In: Experiment@ International Conference, Funchal, Portugal; 2019.
Cameirão MS, Badia SBI, Oller ED, Verschure PFMJ. Neurorehabilitation using the virtual reality based rehabilitation gaming system: methodology, design, psychometrics, usability and validation. J Neuroeng Rehabil. 2010;7:48.
Bermúdez i Badia S. AnTS: analysis and tracking system. 2014. Available from: http://neurorehabilitation.m-iti.org/tools/.
Pendlebury ST, Mariz J, Bull L, Mehta Z, Rothwell PM. MoCA, ACE-R, and MMSE versus the National Institute of Neurological Disorders and Stroke–Canadian Stroke network vascular cognitive impairment harmonization standards neuropsychological battery after TIA and Stroke. Stroke. 2012;43(2):464–9.
Tan HH, Xu J, Teoh HL, Chan BP-L, Seet RCS, Venketasubramanian N, et al. Decline in changing Montreal cognitive assessment (MoCA) scores is associated with post-stroke cognitive decline determined by a formal neuropsychological evaluation. PLoS One. 2017;12(3):e0173291.
Reitan RM. Validity of the trail making test as an Indicator of organic brain damage. Percept Mot Skills. 1958;8(3):271–6.
Cavaco S, Gonçalves A, Pinto C, Almeida E, Gomes F, Moreira I, et al. Trail making test: regression-based norms for the Portuguese population. Arch Clin Neuropsychol. 2013;28(2):189–98.
Wechsler D. Escala de Memória de Wechsler - 3a Edição: Hogrefe Editora; 2008. Available from: https://www.hogrefe.pt/shop/escala-de-memoria-de-wechsler-3-edic-o.html. Cited 2018 Mar 23.
Wechsler D. Escala de Inteligência de Wechsler para Adultos - 3a Edição: Hogrefe Editora; 2008. Available from:. https://www.hogrefe.pt/shop/escala-de-inteligencia-de-wechsler-para-adultos-3-edic-o.html. Cited 2018 Mar 23.
Patchick E, Vail A, Wood A, Bowen A. PRECiS (patient reported evaluation of cognitive state): psychometric evaluation of a new patient reported outcome measure of the impact of stroke. Clin Rehabil. 2016;30(12):1229–41.
Faria AL, Alegria J, Pinho MS, Badia SB. Avaliação feita pelo Doente sobre o seu Estado Cognitivo. 2018. Available from: https://www.click2go.umip.com/i/coa/precis_pt.html.
Brooke J. SUS-A quick and dirty usability scale. Usability Eval Industry. 1996;189:194 Nir-Hadad SY, Weiss PL, Waizman A, Schwartz N, Kizony R. A virtual shopping task for the assessment of executive functions: validity for people with stroke. Neuropsychol Rehabil 2015;27(5):808–833.
Neguţ A, Matu SA, Sava FA, David D. Task difficulty of virtual reality-based assessment tools compared to classical paper-and-pencil or computerized measures: a meta- analytic approach. Comput Hum Behav. 2016;54:414–24.
Parsons TD, Barnett MD. Virtual apartment stroop task: comparison with computerized and traditional stroop tasks. J Neurosci Methods. 2018;309:35–40.
Faria AL, Cameirão MS, Couras JF, Aguiar JRO, Costa D, Martins G, et al. Combined cognitive-motor rehabilitation in virtual reality improves motor outcomes in chronic stroke – a pilot study. Front Psychol. 2018;9. Available from: https://www.frontiersin.org/articles/10.3389/fpsyg.2018.00854/abstract. Cited 2018 May 20.
Wong GK, Mak JS, Wong A, Zheng VZ, Poon WS, Abrigo J, Mok VC. Minimum clinically important difference of Montreal cognitive assessment in aneurysmal subarachnoid hemorrhage patients. J Clin Neurosci. 2017;46:41–4.
Mioshi E, Dawson K, Mitchell J, Arnold R, Hodges JR. The Addenbrooke’s cognitive examination revised (ACE-R): a brief cognitive test battery for dementia screening. Int J Geriatr Psychiatry. 2006;21(11):1078–85.
Duncan PW, Bode RK, Min Lai S, Perera S. Rasch analysis of a new stroke-specific outcome scale: the stroke impact scale. Arch Phys Med Rehabil. 2003;84(7):950–63.
The authors would like to thank Joana Alegria and Carolina Jorge for the collaboration in data collection and Teresa Paulino for the Reh@City v2.0 software development.
This study was supported by the European Commission through the RehabNet project (Neuroscience-Based Interactive Systems for Motor Rehabilitation) under grant 303891 RehabNet FP7-PEOPLE-2011-CIG, LARSyS UID/EEA/50009/2019, MACBIOIDI (INTERREG program MAC/1.1.b/098), FCT through the BRaNT project (PTDC/CCI-COM/31046/2017) and Agência Regional para o Desenvolvimento da Investigação, Tecnologia e Inovação.
Ethics approval and consent to participate
This study was approved by the Madeira Health Service - SESARAM Ethical Committee (approval number 13/2016) and all the participants gave their informed consent.
Consent for publication
Consent for publication of individual data has been obtained from all the participants of the study.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Faria, A.L., Pinho, M.S. & Bermúdez i Badia, S. A comparison of two personalization and adaptive cognitive rehabilitation approaches: a randomized controlled trial with chronic stroke patients. J NeuroEngineering Rehabil 17, 78 (2020). https://0-doi-org.brum.beds.ac.uk/10.1186/s12984-020-00691-5
- Cognitive rehabilitation
- Virtual reality
- Ecological validity