You cannot improve anything if you cannot measure it, – Lord Kelvin
To measure things you need some data that is why companies such as Google and Facebook monitor their user activity every day to gather important activity insights. The gathered data are used to recommend new products or services or to improve business processes.
In data analysis, such approach is called Data-Driven Decision Making (DDDM).
According to Docebo Report 2019, the eLearning community has started applying DDDM using Data Learning Analytics technologies or Learning Management System Reporting.
Data Analytics is a learning technology with high priority for 50% organizations
But, even having access to data analytics or LMS reporting, most Chief Online Officers have a lack of knowledge to process the gathered data. They are unable to apply appropriate techniques and tools to mine learning data and it drives them away from effective use of Data Analytics.
In essence, the gap between effectiveness of data usage and existing/captured data flow is understanding what decisions these data can help make and how it can actually help.
What is data-driven decision-making in online learning?
Data-driven decision-making in learning refers to a continuous cycle of identifying, collecting, combining, analyzing, interpreting and acting upon educational data from different sources to report, evaluate and improve resources, processes and outcomes of organizations.
To describe the DDDM process Rand Education recommends to use the following framework:
The framework suggests that multiple forms of data are first turned into information via analysis and then combined with stakeholder understanding and expertise to create actionable knowledge.
This framework can be interpreted through 4 steps:
1. Data Collection – collection and organization of “raw” educational data* about learners’ activity and their performance.
*“Educational or student-level data refers to any information that educators, schools, districts, and state agencies collect on individual students, including data such as personal information, enrollment information, academic information, and various other forms of data collected and used by educators and educational institutions“
There are several types of data that educators and instructors should use in their data analysis process:
-
Input data – student’s background characteristics
-
Process data – quality/quantity of instructional materials,
-
Outcome data – student’s retention and completion rates
-
Satisfaction data – student’s satisfaction rates
2. Data Analysis – analysis of learner’s data and information to get meaningful knowledge about eLearning courses or programs.
The type of analysis depends on the type of the obtained data, therefore eLearning specialists identify such types of data analysis:
a. Cluster analysis
Cluster analysis is a number of statistical methods of partitioning data into homogeneous parts for classifying the data. It divides these data into meaningful or useful groups known as clusters. Clustering analysis in eLearning deals with the task of how to group students into different clusters. For example, instructors can effortlessly identify student groups with high and low activity using classification and clustering techniques. Learning Management Systems and 3rd party analytics tools include cluster analysis in their reporting tool kit.
b. Descriptive Analysis
Descriptive Analysis is the simplest and the most common form of data analysis. Descriptive analysis answers the “what happened” by summarizing past data basically in the form of dashboards. The biggest use of descriptive analysis in eLearning is to track Time and Engagement Metrics:
-
The average number of actions of the learners
-
Progression of users through the experience (for example, 32% of your learners started just one challenge, 44% started two challenges, and 16% started three or more challenges)
-
Learner’s Retention Metrics
c. Diagnostic Analysis
Diagnostic Analysis is a form of advanced analytics which examines data or content to answer the question “Why did it happen?”, and is characterized by techniques such as drill-down, data discovery, data mining and correlations. For example, diagnostic analysis in an LMS or analytics tool can be presented by heat maps – visualised engagement elements using colours with popular areas in learning content.
d. Predictive Analysis.
Predictive analytics is another way of using eLearning data to create predictions about future student progress, using techniques such as data mining, machine learning and predictive modelling. For example, using past engagement and participation indicators, your LMS system or analytics tool may predict how your students will perform in your present or future eLearning course or program.
3. Data Identification – define new instructional design approach to apply meaningful knowledge. When we speak about the instructional design approach it refers to a framework or process that helps to develop instructional materials in eLearning course. In our course creating: step by step guide we covered all the possible instructional design models and techniques to improve your eLearning course or program.
4. Data Improvement – define questions on how to improve student experience using the collected knowledge.
As you can see, data-driven decision-making in learning is quite a challenging process, but using modern eLearning analytics tools (LMS, MOOCS systems and 3rd party reporting tools) you can simplify the process and data analysis. Therefore you should know what eLearning analytics is and how it could be helpful in DDDM framework steps.
What is called eLearning analytics?
The measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs, – Learning Analytics 2011
eLearning analytics tracks, measures, analyses and reports on the data that learners produce when they engage with your eLearning service. eLearning analytics can also show you which methods are more effective for different student groups. If you unify and set up all your data, you will be able to personalize your service and significantly improve the learning journey of your users. P.S. If you look through the eLearning analytics related literature, you will also find the term “Educational Data Mining” (EDM).
When facing an eLearning analytics project, you should be aware of the model defined by Dr. Mohamed Amid at Aachem University. This model describes the most relevant questions you should ask yourself in order to set up and optimize your eLearning analytics solution:
1. What? What kind of data will eLearning analytics system gather and manage?
As we know, eLearning analytics is a data-driven process, and by definition needs data to provide trainers and content developers with meaningful and actionable metrics. We defined various types of data that could be gathered by the system in the first part of the article.
2. Who? Who is the analysis targeted at? What kind of stakeholders are there?
In an organizational context, a stakeholder is a constituency of an organization (Thompson and Strickland, 2001). In the same way, the stakeholders of eLearning analytics are those who are affected by it and those who will benefit the most from using it.
According to the research “The Pulse of Learning Analytics Understandings and Expectations from the Stakeholders”, there are 3 main stakeholder groups which are engaged in eLearning Analytics:
- Learners
- Instructors
- Educational Institutions
3. Why? What are your objectives? What do you want to see in your reports?
There are 2 categories of eLearning analytics objectives: educational and business. The Educational objective is targeting at improving online learning impact and student’s performance, such as:
- Reducing students’ dropouts
- Improving students’ understanding and learning
- Deciding which content is relevant for a certain user
- Improving training materials
The Business one is targeting at improving the return on investment (ROI) of educational initiatives.
At this stage, the analysis outcome is interpreted in order to achieve the objectives of eLearning analytics.
4. How? How will the system analyse the collected data?
To get a full picture of the impact of your eLearning course or program it is wise to use built-in analytics models such as real time dashboards, surveys, user feedback and other reporting tools.
eLearning analytics is an ongoing process. It doesn’t end once corrective/remedial action is taken. To ensure the effectiveness of the model, you need to make sure the analytics cycle is closed through continuous review and benchmarking.
An eLearning Analytics Cycle considers four parts:
- Learning environment where stakeholders produce data;
- Big data which consist of massive amounts of datasets;
- Analytics which comprises different analytical techniques and metrics;
- Review where objectives are achieved to optimize the learning environment.
In conclusion, according to Dr. Mohamed Amid’s model and analytics cycle, we can say that eLearning Analytics is about obtaining student’s insights from online education data, using data science techniques and having a clear set of educational or business goals in mind.
So, if you wish to learn where, when, and how your learners perform in an online course or a program, you should capture eLearning metrics and launch the reporting process.
Depending on the level of services you are looking for and the budget you are allocated to, introducing an analytics solution in your online learning, you can choose a number of approaches to move forward (LMS, MOOCs systems, 3rd party tools).
The next part of our article will cover all the possible reports and metrics which could be integrated in eLearning analytics system (using LMS case as an example).
LMS Reporting Requirements
Learning Management Systems provide a comprehensive toolkit of analytics and reporting tools that organizations can use to visualize and retrieve valuable data about different areas of online courses.
LMS users have access to different analytics and reporting functionalities that offer meaningful information about student activities. But which set of metrics and reports can boost your working process?
Raccoon Gang team composed an ultimate list of eLearning analytics metrics that could be implemented in your LMS environment.
LMS reporting features and metrics
- “Course/program progress and completions”
- “Course Status” – the current situation of students’ enrollments (the dynamics of students’ enrollment and unenrollment metrics)
- “Number of students who enrolled in a course”
- “Number of students who unenrolled from a course”
- “Total number of students who are currently passing a course”
- “Last access by user” – the last time a user logged into your LMS to take course content. If it has been a long time then you can follow-up accordingly.
- “Total time spent on course/program”
- “Performance grade” – learner’s test/assessments score in an online course or program
- “Current learners location” – this metric tells you exactly where in your online course/program the learner is currently on
- “Learning plan reports”
- “User activity tracking” (# of video views, discussion activities, etc.)
- “Most viewed course parts”
- “Learning Path” – is a road map of learner’s participation in an online course or program
- “Attempts and answers breakdown” – information on the average score and learner’s response distribution for each question/problem
- “Gamification reporting stats” (e.g., badges and contests)
- “Time spent in separate course/program part”
- “Quiz/assessments performance”
- “Individual quiz/assessments answers”
- “Identification of low-performing and high-performing learners”
- “Clustering learners’ activity and characteristics”
We recommend to use the following set of eLearning analytics metrics in your robust LMS reports.
Raccoon Gang developed own analytics tool which turns the metrics mentioned above into easy-to-use reports that are based on diagnostic, cluster and descriptive analysis methods. It is a good case of how modern instructors and course owners can monitor the online progress of their learners through custom reports.
Types of LMS Reports
1. “Enrollment Stats Report” (using data analysis) shows the dynamics of enrollment metrics
The set of analytics metrics reflected in the report:
a. “Course Status”
b. Number of students who enrolled in a course
c. Number of students who unenrolled from a course
d. Total number of students who are currently passing a course
How the report could look:
2. “Learner’s Activity Report” (using data analysis) indicates which parts of your course are the most difficult or interesting for your students:
The set of analytics metrics reflected in the report:
a. “User activity tracking” (# of video views, discussion activities, etc.)
b. “Most viewed course parts”
How the report could look:
3. “Learner’s Progress Report”
The set of analytics metrics reflected in the report:
a. “Performance grade”
b. “User activity tracking” (# of video views, discussion activities, etc.)
How the report could look:
4. “Problem Report” (using diagnostic analysis) shows which parts of a course require improvement, and calculates the ratio of right and wrong answers of students in assessments.
The set of analytics metrics reflected in the report:
a. Attempts and answers breakdown
b. “Quiz/assessments performance”
c. “Individual quiz/assessments answers”
How the report could look:
5. “Progress Funnel Report” (using descriptive analysis) shows a “road map” of learner’s participation in an online course or program
The set of analytics metrics reflected in the report:
a. “Learning Path”
b. “Current learners location”
How the report could look:
6. “Cluster Report” (using cluster analysis) clusters your learners into groups based on their current progress (from low-performers to high-performers)
The set of analytics metrics reflected in the report:
a. “Identification of low-performing and high-performing learners”
b. “Clustering learners’ activity and characteristics”
How the report could look:
Final Words:
LMS reporting and analytics could be helpful on different levels of online learning, including the course, curriculum, institutional, and national level. eLearning analytics can provide insights into what is happening with a learner in nearly real-time. Armed with this information, course owners or instructors can make suggestions to learners that will help them succeed in an online environment.
REFERENCES:
- “Cutting Through the “Data-Driven” Mantra: Different Conceptions of Data-Driven Decision Making”, Rand Education
- “Types of Data Analysis”,
- “What are e-Learning analytics about?“,
- “Learning Analytics: Principles and Constraints”, Originally published in: Khalil, M. & Ebner, M. (2015). In Proceedings of World
- Conference on Educational Multimedia, Hypermedia and Telecommunications 2015. pp. 1326-1336. Chesapeake, VA: AACE.
- “Understanding LMS Reports: 12 Things to Look at in Your eLearning Statistics”.
- What is data-driven decision-making in online learning?
- What is called eLearning analytics?
-
LMS Reporting Requirements
- LMS reporting features and metrics
- Types of LMS Reports
- 1. "Enrollment Stats Report” (using data analysis) shows the dynamics of enrollment metrics
- 2. "Learner's Activity Report" (using data analysis) indicates which parts of your course are the most difficult or interesting for your students:
- 3. "Learner's Progress Report"
- 4. "Problem Report" (using diagnostic analysis) shows which parts of a course require improvement, and calculates the ratio of right and wrong answers of students in assessments.
- 5. "Progress Funnel Report" (using descriptive analysis) shows a "road map" of learner's participation in an online course or program
- 6. "Cluster Report" (using cluster analysis) clusters your learners into groups based on their current progress (from low-performers to high-performers)
