Work

Wynn Mo

Final Year Honours Project @ University of Auckland

Research in collaboration with Danika Chhour

2024

Back to Top

Tangible AR for Data Structure Learning

Data structures are fundamental concepts in science courses. However, learning data structures is challenging - there is an apparent demand for tools to better support students’ learning of such concepts. For my final year honours project, we designed and developed a mobile augmented reality game to help students understand data structures. Users interact with real-world blocks representing nodes of a structure, while virtual, abstract information about the data structure is overlaid on top.

Data structures are fundamental concepts in science courses. However, learning data structures is challenging - there is an apparent demand for tools to better support students’ learning of such concepts. For my final year honours project, we designed and developed a mobile augmented reality game to help students understand data structures. Users interact with real-world blocks representing nodes of a structure, while virtual, abstract information about the data structure is overlaid on top.

How augmented reality fills the gap in data structures education?

As part of our honours project, we read multiple published research papers on related work which include challenges in data structures education, and current approaches to overcoming these challenges in data structures education.

As part of our honours project, we read multiple published research papers on related work which include challenges in data structures education, and current approaches to overcoming these challenges in data structures education.

Research Intent

Researching published papers in data structures education helped us define the following research question:


“Whether a tool combining tangible real world objects with virtual augmentation helps students better understand and engage with data structures.”


With this research intent, we embraced multi modal pedagogy to fill the gap in data structure education by allowing students to touch physical representations of data structures while simultaneously seeing abstract information overlaid.

Researching published papers in data structures education helped us define the following research question:


“Whether a tool combining tangible real world objects with virtual augmentation helps students better understand and engage with data structures.”


With this research intent, we embraced multi modal pedagogy to fill the gap in data structure education by allowing students to touch physical representations of data structures while simultaneously seeing abstract information overlaid.

Linked list data structure

Design - Game Flow

This project helps students understand singly linked lists, which are often misunderstood due to abstract pointers, lack of real-world context, and confusion from prior array knowledge [2].

These tasks require managing multiple pointers, especially the tail, which is hard to visualise. To address this, we designed a simple game flow with training and play modes to guide students through these tricky concepts.

This project helps students understand singly linked lists, which are often misunderstood due to abstract pointers, lack of real-world context, and confusion from prior array knowledge [2].

These tasks require managing multiple pointers, especially the tail, which is hard to visualise. To address this, we designed a simple game flow with training and play modes to guide students through these tricky concepts.

[2] Daniel Zingaro, Cynthia Taylor, Leo Porter, Michael Clancy, Cynthia Lee, Soohyun Nam Liao, and Kevin C. Webb. 2018. Identifying Student Difficulties with Basic Data Structures. In Proceedings of the 2018 ACM Conference on International Computing Education Research (ICER '18). Association for Computing Machinery, New York, NY, USA, 169–177.

Design - User Experience

To help users understand abstract linked list concepts, the AR tool displays simple visual overlays on top of the wooden blocks (Figure 1). These overlays show the node’s outline, data value, pointers to the next node, and clearly labelled head and tail pointers.



Text box prompts (Figure 2) provide soft guidance without giving step-by-step instructions. They remind users of:

The gesture needed

The goal of the task


This encourages users to apply their own knowledge and reinforces learning.

To help users understand abstract linked list concepts, the AR tool displays simple visual overlays on top of the wooden blocks (Figure 1). These overlays show the node’s outline, data value, pointers to the next node, and clearly labelled head and tail pointers.



Text box prompts (Figure 2) provide soft guidance without giving step-by-step instructions. They remind users of:

  • The goal of the task

  • The gesture needed


This encourages users to apply their own knowledge and reinforces learning.

Figure 1: Visual overlay illustrating the selected node

Figure 2: Example of a prompt to guide user towards removing node at the tail of the list

Design - User Interaction

The tool uses coloured wooden blocks to represent linked list nodes and hand gestures for interaction, creating a multi-modal learning experience.

The tool uses coloured wooden blocks to represent linked list nodes and hand gestures for interaction, creating a multi-modal learning experience.

  • Users can move blocks to show that linked lists are connected by pointers and not stored in order.

  • To add or remove nodes, users move blocks into or out of the screen’s view.

  • To traverse the list, users tap on each block in sequence, showing that linked lists must be accessed one node at a time.

These simple gestures are designed to be intuitive, helping users focus on learning without being distracted by complicated controls.

These simple gestures are designed to be intuitive, helping users focus on learning without being distracted by complicated controls.

Implementation

We designed and coded the tool using mobile augmented reality (AR), combining physical interaction with virtual overlays. The application runs on an iPad Pro for a larger interface, but is also compatible with other iOS and Android devices through Unity’s AR Foundation. A tablet stand is used to free up both hands for interaction.

We built the app in Unity, deployed it using Xcode for iOS, and integrated OpenCV for Unity to enable:

We designed and coded the tool using mobile augmented reality (AR), combining physical interaction with virtual overlays. The application runs on an iPad Pro for a larger interface, but is also compatible with other iOS and Android devices through Unity’s AR Foundation. A tablet stand is used to free up both hands for interaction.

We built the app in Unity, deployed it using Xcode for iOS, and integrated OpenCV for Unity to enable:

  • Colour tracking of the wooden blocks

  • Hand pose estimation to detect gestures like tapping and movement

Gameplay example of removing a tail from a linked list

Evaluation Methodology
User Study Design

We compared our tangible AR tool with an existing data structures browser game called the VisuAlgo.

We compared our tangible AR tool with an existing data structures browser game called the VisuAlgo.

  • VisuAlgo: Shows animations of linked list operations

  • Tangible AR tool: Interactive tutorial with gesture-based tasks and a warm-up phase

To prevent bias, the tool order and test questions were rotated. Each of the participants:

  1. Completed a pre-study test and questionnaire

  2. Used both tools for 5–10 minutes each

  3. Completed a test and questionnaire after each tool

  4. Finished with a post-study questionnaire

VisuAlgo interface

Demographics

In total, 21 students completed the user study.

→ 90% of the participants were either 21 or 22 years old, corresponding to third or fourth year undergraduate students.

→ 9 of the 21 participants had never done any data structures before, whereas the others had done at least one course containing a data structures module.

The data structure modules that students had completed varied across engineering disciplines, therefore, there is likely a difference in the depth and method in which they were taught.

Competency Tests and Questionnaires

Our tests and questionnaires were designed to give us insight on specific measures:

  1. Understanding

  2. Engagement and Motivation

  3. Interest and Enjoyment

  4. Value and Usefulness




Our tests and questionnaires were designed to give us insight on specific measures:

  1. Understanding

  2. Engagement and Motivation

  3. Interest and Enjoyment

  4. Value and Usefulness

Results - Understanding

Each answer was marked on a scale of 0 to 4, depending on which elements of the model answer the user included. For this measure, I have shown the results of one of the three questions as an example.

50% of participants completing this question after using the tangible AR tool got full marks compared to 14% of participants completing this question after using VisuAlgo.

1. How do you represent a linked list in a graphical view?

Results - Engagement and Motivation

Notably, no respondents strongly disagreed or disagreed with the statement "I found the tool engaging" concerning the tangible AR tool.

In contrast, 20% of participants (5% "Strongly Disagree," 5% "Disagree," 5% "Slightly Disagree," and 5% "Neutral") provided more negative or neutral responses towards the VisuAlgo tool. These results indicate that the tangible AR tool provides more favourable intrinsic engagement and motivation.

Results - Interest and Enjoyment

Notably, the tangible AR tool received high scores on the Likert scale in response to the statement "I would describe this tool as very interesting", which indicates that participants were interested in the novel tool.

These findings suggest that, in this context, both the tangible AR tool and VisuAlgo effectively make the learning process enjoyable and interesting, which is a key contributing factor to student success.

1. How do you represent a linked list in a graphical view?

Results - Value and Usefulness

One participant noted that "[the tools] worked well together", which is also consistent with literature findings that students are more likely to succeed by combining different tools and representations of concepts to reinforce their mental models.

Reflection and Future Work

One signicant restriction of the tangible AR tool is that the tracking of colours and hands is limited by the OpenCV library used and, therefore, requires a bit of tedious set-up to ensure the colours are tracked well throughout the tutorial. Upgrades to the tracking library will likely result in better user experiences and enjoyment and engagement with the tool.

The findings in this work have the potential to help some students overcome common barriers and challenges in computer science data structure courses.

This research project was picked up by incoming final year engineering students to further develop more game modes for the tool.

Poster Design