Final report of DISA Seed project: An Exploration of the Challenges and Possibilities of Multidimensional Visualization in the Context of Visual Learning Analytics

Final report of DISA Seed project:

An Exploration of the Challenges and Possibilities of Multidimensional Visualization in the Context of Visual Learning Analytics

Rafael M. Martins

, Marcelo Milrad

, and Italo Masiello

1

Dept. of Computer Science and Media Technology /

Dept. of Pedagogy and Learning Linnaeus University

1. Summary of Activities and Results of the Seed Project

We report on 8 main groups of activities (A1-8) and 7 main results (R1-6) performed and obtained during the period of the Seed project, as described next. After a couple of initial planning meetings at LNU (A1), we proceeded to travel to Gothenburg and meet with key members of the managing and data team of Hypocampus (A2), where we discussed the contents of their data and the possibilities of collaboration. Some time was spent for loading, studying, cleaning, and understanding their data, resulting in an exploratory data analysis report (R1).

With the new knowledge, more concrete goals for the collaboration were defined and discussed online, both asynchronously in a Slack room created for the specific purpose of discussing learning analytics between the members of the project and in a Zoom meeting with the Hypocampus data team. After an initial non-systematic analysis of recent and relevant papers, a systematic literature review of the use of t-SNE in the context of learning analytics has been planned and partially performed during the project (A3, R2), and is still ongoing. This, coupled with the data and the cooperation of the company, was enough to inform the development of a prototype of a visual analysis system for investigating multidimensional student behavior extracted from the company’s data (A4, R3). The first stable version of the prototype was discussed both internally and with Hypocampus, again in person in their Gothenburg office. The results obtained with the improved prototype were described in a short paper published in the proceedings of EC-TEL 2019, the 14th European Conference on Technology-Enhanced Learning (A5, R4), as a LNCS Springer volume

. This publication also included the presentation of the prototype at the conference in Delft, NL (A6). In parallel, the work of a summer intern was also funded and supervised (A7), which resulted in the development of a second prototype (R5) with a different focus. Finally, using all the experience gathered during the project, the team members collaborated on writing and submitting a proposal for external research funding (A8, R6) from VINNOVA (Ansökan till Utmaningsdriven innovation - Steg 1) including partners such as RISE Research Institutes of Sweden, Studi AB, Kungsbacka kommun, Skellefteå kommun, NTI-gymnasierna, Eskilstuna kommun, Lidingö Stad and Västerviks kommun.

2. To what extent have the objectives of the project been achieved / challenges were met?

Explain the outcome and indicate the reasons for possible deviations.

The submitted proposal had three main objectives, which are discussed in details below. Two main deviations are worth mentioning: first, the practical perspective (O2) was prioritized over the theoretical one (O1) during the project, mainly due to the fact that the collaboration with the company proceeded in a much smoother and more productive way than predicted in the proposal. Since we got a very active cooperation from the data team of Hypocampus in understanding the data and its possible uses, we decided to move on to building, validating (and eventually publishing) a much more interesting and complex prototype than initially planned. This re-allocation of resources meant we did not fully reach the objective of having a complete draft of a systematic literature review by the end of the project, but we believe that the final balance was positive and the available human resources were better used this way. Second, due to strategic reasons, we changed the plan from submitting a KK proposal

1

Martins R.M., Berge E., Milrad M., Masiello I. (2019) Visual Learning Analytics of Multidimensional Student Behavior in Self-regulated Learning. In: Scheffel M., Broisin J., Pammer-Schindler V., Ioannou A., Schneider J.

(eds) Transforming Learning with Meaningful Technologies. EC-TEL 2019. Lecture Notes in Computer Science,

vol 11722. Springer, Cham

to submitting a VINNOVA proposal instead. According to the acquired experiences of the Seed team and the profile of the external partnerships obtained during the period, we decided that the Ansökan till Utmaningsdriven innovation - Steg 1 was a better fit for a follow-up of the project.

The first objective (O1) was a theoretical one, where we intended to understand the potential open challenges in learning analytics where multidimensional visualization might offer significant contributions. Out of all the possible multidimensional visualization techniques that are available, we chose to focus on t-SNE due to its position as a state-of-the-art dimensionality reduction method, its current popularity in many different domains of research, and the familiarity and expertise of the team members in its application. Initially, a non-systematic analysis of recent and relevant papers (from the most important learning analytics venues) was performed in order to identify gaps that could be exploited. The results became part of the short paper published in the proceedings of the EC-TEL 2019 conference. This analysis is currently being extended as a systematic literature review, performed according to a more formal protocol for searching, gathering, and analyzing the papers related to keywords such as “learning analytics” and “t-SNE”. The full set of papers to be included is already gathered, a pilot analysis was performed to validate the categories and the protocol for extracting data from the papers, and the full analysis is now on the way. We intend to submit an initial summary of results as a poster to LAK’20 (https://lak20.solaresearch.org/), one of the best conferences in learning analytics, and then the full paper to a journal in 2020.

The second objective (O2) was a practical one, where we intended to familiarize ourselves with the large-scale, multidimensional data provided by our industrial partners, and to obtain initial insights about its structure and potential. The company (Hypocampus) was very quick to provide the data to us and helped us very actively in understanding it by providing also a kind of “user manual” to understand all the fields and entities it contained.

After an exploratory data analysis report by our team, and a second discussion with the company about our common research goals, we settled on using t-SNE to explore the data as a set of student vectors, which are basically multidimensional embeddings of students based on their scores on the exercises of the Hypocampus mobile app. The main novel contributions of the work were (a) the concept of the student vectors, their meaning, relevance, and structure; (b) the actual data mining process of extracting, cleaning, and formatting the data in order to generate the student vectors; and (c) the method of exploration of the t-SNE projection of the student vectors in order to not only find groups of students with similar profiles, but also to determine what are the important features that separate such groups from the rest. A short paper, describing the process and results, was published in the proceedings of the EC-TEL’19 conference, and the prototype was presented in Delft, NL. The source code is available at https://gitlab.com/rmmartins/ec-tel-2019. In parallel, the second prototype developed by the summer intern went in a different direction, but also based on the experience obtained from the interactions with the company. The goal of this work was to start bridging the gap between the complex, multidimensional data that arises from learning contexts (and potentially many other contexts as well) and the end users who need a quick, easy, and customizable dashboard to analyze that. The intern developed the first version of a prototype of a framework that can instantiate a simple dashboard directly from the data itself, including initial suggestions of visualizations for each variable of the data depending on their type. The views are connected and cross-filtered interactively, enabling an initial workflow of investigation of the underlying multidimensional data. The current version is running at https://crossboard.netlify.com/. We intend to proceed with the development and validation of this framework in the near future. In conclusion, we believe this objective (O2) was not only reached but expanded beyond the initial expected results.

The third and final objective was to consolidate all the acquired knowledge from all the different sources and to

submit a proposal for external research funds in order to extend the work. This was accomplished with the

submission of a VINNOVA proposal for the call Ansökan till Utmaningsdriven innovation - Steg 1. The objective

of the submitted proposal is much more ambitious and far-reaching than what was performed during the Seed

proposal: to build a strong and empirically-supported model (and a reference implementation) for data-driven

exam-free assessments of students. The proposal involves many partners that we communicated with during the

Seed period (but were not directly involved in the Seed project), such as companies that work with educational

platforms, researchers in educational technology, and public institutions in many different municipalities. We

bring from the experience of the Seed project the perspectives on how visual learning analytics and multidimensional visualization techniques (such as t-SNE) can help with achieving this goal, since such a model/platform will invariably include a strong component of Human-Computer Interaction where teachers will analyze students’ performances from as many perspectives (and as varied) as possible. All the Seed team members are involved in the proposal in important research leadership roles.

4. Has the budget of the project been followed? If not, explain deviations and how these have affected the project.

The total cost of the DISA-financed budget was respected, with a few small deviations. First, the planned collaboration of Alisa Sotsenko Lincke was not concretized, due to no special reasons other than that the project developed and proceeded well without the need of an extra actor. The costs planned for her time were absorbed by the extra time used by the main author of the proposal, by the costs of attending the EC-TEL’19 conference, which were not anticipated in the initial proposal, and by a small correction of the hours planned to the summer intern, which were initially 90 but he worked 50% for 5 weeks (so 100 hours in total). The budget for the self- funded hours, however, proved to be quite inaccurate, as the members of the project spent at least 50% more than what was planned. These deviations are highlighted in the attached spreadsheet.

5. General description of the experience gained in the project and an assessment of project results, including any unexpected effects / events.

This Seed project was responsible for many more outcomes than what is objectively discussed in previous sections as direct results of the project. The most important of them was to bootstrap the active collaboration between the members of the team, which resulted in the first steps towards establishing a research environment in learning analytics. We have identified many research interests in common related to data-driven solutions and human- computer interaction, and the team includes a diverse set of expertise profiles that have so far resulted in productive cooperative work. Examples of unexpected effects and events that resulted from the Seed project were:

(a) the submission of a research funding proposal to the Wallenberg Foundations; (b) the submission of a research funding proposal to Vetenskapsrådet (VR); (c) ongoing contacts and meetings with many large companies that work with educational technologies; and (d) the start of a new PhD student (that has started her doctoral studies in August 2019) co-supervised by all the three team members of the Seed project.

6. Describe the next steps.

Concretely, there are currently three main avenues of continued work for the members of the project: (1) the co-

supervision of the new PhD student in Computer and Information Systems, who is working with research on

visual learning analytics; (2) the work on the VINNOVA project, which will hopefully be granted and will start

soon; and (3) the knowledge and experience gained in this seed project will be used as the basis of the content to

be taught in a new master course on learning analytics that will start 15-18 months from now. Avenues (1) and

(2) are related, since the PhD candidate’s research is also moving towards the establishment of interactive visual

analysis methods for the assessment of students through multiple different perspectives. Besides these three main

avenues, we intend to keep on working together to make the research environment on learning analytics stronger,

and to keep pushing the current ongoing talks with companies in order to turn them into concrete research results.