This report presents the findings of an evaluation of Arility, an augmented reality road safety training package developed for children. The system presents the user with a variety of scenarios and then asks them to consider what they feel the character in the scenario should do to maintain their safety. In the UK version there are 3 scenarios for Key Stage 1 (KS 1) children and 3 for Key Stage 2 (KS 2) children.
Method
Due to Covid, and difficulties in encouraging schools to participate in the evaluation process over that period, the evaluation faced a range of challenges. It was therefore only possible to allocate schools to either a control or treatment group based on their willingness to participate. At both KS 1 and 2 the Treatment Group (TG) was formed of 3 schools, with 2 schools forming the Control Group (CG). The schools were drawn from Kent, Warwickshire and Wirral. We would like to thank the Road Safety Officers from those areas for their support, it would not have been possible to complete this evaluation without them. KS 1 children completed a specially developed knowledge assessment test at 2 time points (T1 and T2) approximately 4 weeks apart, with the treatment group receiving the intervention approximately 2 weeks after the initial assessment. KS 2 children received a more complex knowledge assessment but also completed a Virtual Reality (VR) skills assessment. The Road Safety Officers (RSO) who delivered the sessions were also asked to provide written feedback on the sessions.
Results
Key Stage 1 The situation was less clear for KS 1 where no clear benefits could be identified. The most likely explanation for this finding is that the content for this age group is already being covered in other ways. All 3 of the road safety officers who delivered these sessions had some concerns about the usefulness of the system for this key stage relating to preventing the technology from becoming a distraction and maintaining the children’s focus on the learning outcomes. These concerns could explain the lower scores for enjoying the lesson with 76% indicating they were happy, or very happy, with the lesson and 77% indicating they had enjoyed, or enjoyed it a lot, using the tablets. Both figures are lower than those given by the KS 2 children. It is therefore suggested that the system is only used with the oldest children in KS1.
Key Stage 2 There is clear evidence which demonstrates that Arility did improve KS 2 children’s road safety knowledge, with the knowledge assessment finding a significant improvement (p < .05) between groups at T2. No difference was found at T1 between groups. In particular, the new model commission by Road Safety GB relating to the use of pedestrian crossings appears to have had a positive impact with a significant improvement (p <.05) being identified between the treatment and control group scores at T2 on the set of items specifically testing this area.
The VR skills assessment also found a significant improvement (p < .05) in the children’s ability to select a safe time to cross the road, with 100% of the treatment group selecting a safe time to cross compared to only 82% of the control group. No significant differences were found for the number of times the children looked right or left while crossing a road, or for observing driveways whilst walking along a footpath. However, it should be noted that both groups scored high on both these tasks initially allowing little room for improvement.
The feedback from the KS 2 children was very positive with 94% of the children saying they were happy, or very happy, with the session and 94% saying they enjoyed, or enjoyed it a lot, using the tablet.
The feedback from the RSOs who delivered the training was also positive and supported the findings of the evaluation for the age group. The RSO suggested that they felt that Arility would be most effective with the younger end of KS 2 age range. The RSO believed that it could be used by a non-road safety specialist, but clear guidance notes should be made available.
Technology
Technology was also identified as an issue. Delivering this evaluation was challenging due to Covid but also due to difficulties in finding schools willing to install the Arility app. This difficulty is not one of Arility’s making, but how schools regulate their IT systems. These systems are tightly controlled making it difficult for the class teachers to add new software.
The only way we were able to conduct the evaluation was through the purchase of 26 tablets, but even this approach had challenges, with the trainers experiencing difficulties accessing the schools’ Wi-fi systems which limited access to Arility’s classroom management tools. Therefore, if road safety teams want to utilise the system, they may have to purchase their own tablets, but this raises issues of cost, maintenance, and safeguarding.
