The comparison of teaching efficiency between massive open online courses and traditional courses in medicine education: a systematic review and meta-analysis

Introduction

Although higher education today faces many challenges, including concerns about its role in society, fragmented functions within universities, and growing diversity of the student population, Massive Open Online Courses (MOOCs) probably could probably provide free Internet-based learning opportunities to improve the current situation (1). They can be regarded as part of the Open Educational Resources (OER) movement because they can provide free educational assistance to anyone in any location (2).

Medicine is traditionally slower in the adaptation of new trends, but it is inspired by these new educational patterns as well (3). For example, some systematic reviews demonstrated that many health related MOOCs were conducted (4). The wide appeal of MOOCs in medical education is attributed to their ability to break barriers to education and provide high quality teaching resources to all learners around the globe without a financial cost (5). MOOCs are also seen as a tool in curriculum reform, which requires students to engage in self-directed online preparations prior to face-to-face learning activities (3). Since the start of the implementation of MOOCs in medical education, many benefits can be enjoyed by medical students and continuing medical education (CME) or continuing professional development (CPD) for graduate or postgraduate learners, considered as the main areas of progress in education (6-8).

However, some educators are concerned that the massive number of enrolled students on MOOCs causes difficulty for instructors to implement effective teaching and learning strategies and the emphasis on technology and logistics may broadly overwhelm teaching and learning (9). It is still controversial if MOOCs are the best pattern in medical education for the diverse populations and countries with different economic development levels of the world. Many educators also want to know whether MOOCs can replace traditional medical education or only act as a complementary tool.

Therefore, a systematic review and meta-analyses are required to resolve these questions with definitive analysis providing stronger evidence for choosing a specific educational pattern to teach different medical students from diverse backgrounds. For this reason, we performed a meta-analysis of pooled data from existing retrospective cohort studies to evaluate the passing rates of MOOCs and Traditional Courses.

Methods

Literature search strategy

We carried out and reported this systematic review and meta-analyses according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (10). The published reports were identified and selected by searching the following databases: Science Direct, PubMed, Springer Link and Wiley Online Library, from their date of inception to December 2016 with no lower date limit applied. The search used combinations of the following text strings (MOOC* OR ‘Massive Open Online Course*’) and ‘Medicine Education’. The searches were restricted to English although there was no restriction on origin. We also reviewed the reference lists of all retrieved studies to identify any potentially related articles. All retrieved studies and many potentially relevant articles were carefully read, including the names of authors and the time of publishing, to avoid duplication of data.

Selection criteria

Studies were selected for inclusion in the meta-analyses based on the following criteria: (I) studies using retrospective cohort methods to conduct research on Massive Open Online Courses and Traditional Courses; (II) the MOOCs disciplines were based on the medical profession; (III) the passing rates regarding the effectiveness of Massive Open Online Courses and Traditional Courses were available; (IV) the studies were documented in English. Exclusion criteria for these meta-analyses were as follows: (I) students who participated in the studies and who only enrolled in either MOOCs or Traditional Courses; (II)review articles or case reports; (III) letters, editorials, and expert opinions without original data; (IV) the passing rates of students of MOOCs and Traditional Courses that were not analysed separately in the reports; (V) studies that lacked control groups and did not clearly report the outcomes of interest.

Data extraction and quality assessment

Two reviewers, QJ Zhang and Y Zhou, independently selected the eligible studies and performed data extraction according to a standard protocol. All the data were extracted from four eligible studies (11-14). When the reviewers initially disagreed, it was resolved by discussion whereby a consensus was eventually reached. According to a standard protocol, data comprising several necessary characteristics were extracted: the first author(s) or the name of the study group, the journal the article had been published in, the year of the publication, the country of the study, the number of the students enrolled and the passing rates of students from MOOCs and Traditional Courses. When data were missing or unclear in a paper, the corresponding authors were contacted through mail or email to obtain the necessary information. An article was excluded if there was no response after two contact attempts.

Statistical analysis

The Review Manager 5.3 and STAT 12.0 software was used for the statistical analysis of studies. The OR and 95% CI were used to present the statistical values derived from the efficacy analysis for dichotomous variables. All statistical assessments were 2-sided and the significance level was defined as P<0.05. The fixed effects model was adopted for the pooled analysis if a statistical homogeneity existed among the studies (P>0.05, I²<50%) and the random effects model was utilised for the analysis if a statistical heterogeneity existed among the studies (P<0.05, I²>50%).

Results

Search results and trial characteristics

A total of 383 studies were identified by searching database. Out of all the studies, 34 were identified by searching the PubMed database, 135 by searching the Science Direct database, 99 by searching the Springer database and 115 by searching the Wiley database. We identified 372 studies after 11 duplicates were removed. By scanning titles and abstracts, we excluded 358 studies including 11 reviews and 347 studies with no apparent relevance to the present study. In total, 14 studies were therefore included in the next round of review. After reading the full text of these articles, we removed 10 studies not containing enough data to be analysed. The diagram in Figure 1 represents the flow of identification and inclusion of trials, as recommended by the PRISMA statement. As a result, four studies (11-14) that included a total of 17,339 students were selected for meta-analyses (Table 1). The quality of eligible studies were assessed using Newcastle-Ottawa Quality Assessments Scale (Table 2).

Figure 1 Flow diagram showing the process used for selecting articles.

Table 1 Main characteristics of individual studies included in the meta-analysis
Full table

Table 2 Newcastle-Ottawa Quality Assessments Scale
Full table

Comparison of passing rates between MOOCs and traditional courses

A total of four papers reported results that were simultaneously divided into two groups (MOOCs group and Traditional Courses group). We analysed the passing rates of all students with MOOCs and Traditional Courses. The homogeneity test on these 4 studies resulted in P<0.00001 and I²=99% (Figure 2). There was significant heterogeneity in the analysis of these 4 studies; so, we performed subgroup analysis according to sample size and found that homogeneity test on three studies resulted in P=0.71 and I²=0% if we excluded the study with a big sample size by author Magana-Valladares (Figure 3). An analysis of these 3 studies using the fixed effects model showed an OR of 1.16 (95% CI: 0.90–1.51), although this was not statistically significant (P=0.26), indicating that there are no significant differences in the passing rates of all students of MOOCs and Traditional Courses (Figure 4).

Figure 2 Forest plot of passing rates of MOOCs vs. traditional courses in four studies. MOOCs, Massive Open Online Courses; OR, odds ratio; CI, confidence interval.

Figure 3 Subgroup analysis of passing rates of MOOCs vs. traditional courses according to sample size. MOOCs, Massive Open Online Courses; OR, odds ratio; CI, confidence interval

Figure 4 Forest plot of passing rates of MOOCs vs. traditional courses in three studies. MOOCs, Massive Open Online Courses; OR, odds ratio; CI, confidence interval.

Comparison of passing rates of MOOCs and Traditional Courses within different countries

Regarding the subgroup analysis of three papers according to different professions, we found that the homogeneity test of subgroup analysis on these 3 studies resulted in P=0.42 and I²=0%. An analysis using the fixed effects model showed an OR of 1.23 (95% CI: 0.92–1.65) in American learners and 0.95 (95% CI: 0.55–1.65) in British learners, although, again, this was statistically significant neither in the American (P=0.17) nor in the British case (P=0.86), indicating that there are no significant differences in the passing rates of students from different countries of MOOCs and Traditional Courses. However, we found that the OR of 1.23 in American learners is more than that of British at 0.95, especially in the study from author Schneider et al. in 2016, which showed an OR of 1.27 (95% CI: 0.80–2.01) indicating that the passing rates of all students of MOOCs in America are probably greater than those of Traditional Courses (Figure 5).

Figure 5 Subgroup analysis of passing rates of MOOCs vs. traditional courses according to different countries. MOOCs, Massive Open Online Courses; OR, odds ratio; CI, confidence interval.

Comparison of passing rates of MOOCs and Traditional Courses for different professions

The subgroup analysis of three papers according to different professions, revealed that the homogeneity test of subgroup analysis on these 3 studies resulted in P=0.65 and I²=0%. An analysis using the fixed effects model showed an OR of 1.27 (95% CI: 0.80–2.01) in the physician group and 1.11 (95% CI: 0.81–1.53) in the student group, although, again, this was statistically significant neither in the physician group (P=0.31) nor in the student group (P=0.50), indicating that there are no significant differences in the passing rates of students of different professions of MOOCs and Traditional Courses. However, we found that OR of 1.27 in the physician group and 1.11 in the student group are both more than one and the OR of 1.27 in the physician group is more than that of 1.11 in the student group (Figure 6).

Figure 6 Subgroup analysis of passing rates of MOOCs vs. traditional courses according to different profession. MOOCs, Massive Open Online Courses; OR, odds ratio; CI, confidence interval.

Discussion

‘Connectivism and Connective Knowledge’ in 2008 at the University of Manitoba is considered the first MOOC (15). Unlike traditional online courses that rely mainly on resources posted by the facilitators through a learning management system, this course was accessible on the Internet and open to registration generally without limits on numbers (16). The advantages of MOOCs are that they can offer free educational opportunities to anyone in any location.

The potential value of the growing online phenomenon of MOOCs offers new possibilities to support educational activity (17). In medicine, MOOCs is used to support postgraduate medical training and CPD is being recognised (4,18,19) and there is evidence to demonstrate that doctors and undergraduate students are participating in this educational trend (20).

Some investigators have argued that the size of the electronic online learning MOOCs forum can inhibit some students’ potential involvement (21,22). Others showed that a lengthy course and a static online one-way instructional approach are other disadvantageous factors (23-26). Therefore, it is apparent that controversies still existed about whether MOOCs can replace traditional medical education or only complement it.

Our meta-analysis included four retrospective cohort studies, according to the inclusion criteria. After the analysis of these studies, only three studies are included in the final analysis because one has a big sample size, which produces the significant heterogeneity. Our results showed that there are no significant differences in the passing rates of MOOCs and Traditional Courses (OR =1.16, 95% CI: 0.90–1.51 P=0.26). From these results, we concluded that generally, MOOCs have no advantages over Traditional Courses regarding teaching efficiency, although, we found that the OR is 1.16. However, if a detailed analysis had not been performed, some evidence would have been neglected. For this reason, we used subgroup analysis to find out the passing rates of MOOCs and Traditional Courses within different countries and professions. Our results showed that there are no significant differences in American (P=0.17) and in British (P=0.86). However, we found that the OR of 1.23 in American courses is more than 0.95 in British, indicating that an advantage of MOOCs will gradually appear in America when universities there start investing more money in online education and construct a more extensive network platform. Our results also showed that there are no significant differences in the physician group (P=0.31) and the student group (P=0.50). However, we found that an OR of 1.27 in the physician group is more than that of 1.11 in the student group, indicating that MOOCs are probably more suitable for people with sufficient medical knowledge, such as qualified physicians than students who were still accumulating medical knowledge. With the development of a network platform and online education, we can conclude that MOOCs will probably play an important role in postgraduate medical training in the near future.

It is worth noting that this meta-analysis suffers from several limitations. First, evidence comparing the passing rates of MOOCs with Traditional Courses is still insufficient: only 4 studies were enrolled which makes the results less convincing. Second, high heterogeneity was detected limiting the value of this meta-analysis in some way. Third, other factors, such as economy level and educational level, is likely to affect the passing rate. Owing to lack of data, subgroup analysis can’t be conducted to further explore the association between those factors. Despite above limitations, the value of this meta-analysis comparing the passing rates of MOOCs with Traditional Courses can’t be denied.

Conclusions

The results of our meta-analysis indicated that there was no statistically significant difference in the passing rates of MOOCs and Traditional Courses. However, it is still necessary for universities to invest in online education to promote the development of MOOCs, which will probably have advantages over Traditional Courses for postgraduate medical education in the near future.

Acknowledgements

Funding: This work was supported by the research projects of the social science and humanity on Young Fund of the ministry of Education (Grant number: 15YJCZH061).

Footnote

Conflicts of Interest: The authors have no conflicts of interest to declare.

References

1. Emanuel EJ. Online education: MOOCs taken by educated few. Nature 2013;503:342. [Crossref] [PubMed]
2. Parkinson D. Implications of a new form of online education. Nurs Times 2014;110:15-7. [PubMed]
3. Kononowicz AA, Berman AH, Stathakarou N, et al. Virtual Patients in a Behavioral Medicine Massive Open Online Course (MOOC): A Case-Based Analysis of Technical Capacity and User Navigation Pathways. JMIR Med Educ 2015;1:e8. [Crossref] [PubMed]
4. Liyanagunawardena TR, Williams SA. Massive open online courses on health and medicine J Med Internet Res 2014;16:e191. review. [Crossref] [PubMed]
5. Mayfield CH, Ohara PT, O'Sullivan PS. Perceptions of a mobile technology on learning strategies in the anatomy laboratory. Anat Sci Educ 2013;6:81-9. [Crossref] [PubMed]
6. Lau KH. Computer-based teaching module design: principles derived from learning theories. Med Educ 2014;48:247-54. [Crossref] [PubMed]
7. Volandes AE, Kennedy WJ, Davis AD, et al. The new tools: What 21st century education can teach us. Healthcare 2013;1:79-81. [Crossref] [PubMed]
8. Hoy MB. MOOCs 101: an introduction to massive open online courses. Med Ref Serv Q 2014;33:85-91. [Crossref] [PubMed]
9. Goldberg LR, Bell E, King C, et al. Relationship between participants' level of education and engagement in their completion of the Understanding Dementia Massive Open Online Course. BMC Med Educ 2015;15:60. [Crossref] [PubMed]
10. Moher D, Liberati A, Tetzlaff J, et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. J Clin Epidemiol 2009;62:1006-12. [Crossref] [PubMed]
11. Frank E, Tairyan K, Everton M, et al. A test of the first course (Emergency Medicine) that is globally available for credit and for free. Healthcare 2016;4:317-20. [Crossref] [PubMed]
12. Magana-Valladares L, Gonzalez-Robledo MC, Rosas-Magallanes C, et al. Training Primary Health Professionals in Breast Cancer Prevention: Evidence and Experience from Mexico. J Cancer Educ 2018;33:160-6. [Crossref] [PubMed]
13. Schneider M, Evans R, Haas M, et al. The effectiveness and feasibility of an online educational program for improving evidence-based practice literacy: an exploratory randomized study of US chiropractors. Chiropr Man Therap 2016;24:27. [Crossref] [PubMed]
14. Bowen WG, Chingos MM, Lack KA, et al. Interactive Learning Online at Public Universities: Evidence from Randomized Trials. 2012. Available online: https://sr.ithaka.org/wp-content/uploads/2015/08/sr-ithaka-interactive-learning-online-at-public-universities.pdf
15. Liyanagunawardena TR, Adams AA, Williams SA. MOOCs: a Systematic Study of the Published Literature 2008-2012. International Review of Research in Open and Distance Learning. Available online: 2013;14:201-27. http://www.irrodl.org/index.php/irrodl/article/view/1455/2531
16. Siemens G. Connectivism-A learning theory for the digital age. International Journal of Instructional Technology and Distance Learning. 2005;2:3-10.
17. Rocha-Pereira N, Lafferty N, Nathwani D. Educating healthcare professionals in antimicrobial stewardship: can online-learning solutions help? J Antimicrob Chemother 2015;70:3175-7. [PubMed]
18. Subhi Y, Andresen K, Rolskov Bojsen S, et al. Massive open online courses are relevant for postgraduate medical training. Dan Med J 2014;61:A4923. [PubMed]
19. Harder B. Are MOOCs the future of medical education? BMJ 2013;346:f2666. [Crossref] [PubMed]
20. Aboshady OA, Radwan AE, Eltaweel AR, et al. Perception and use of massive open online courses among medical students in a developing country: multicentre cross-sectional study. BMJ Open 2015;5:e006804. [Crossref] [PubMed]
21. Baggaley J. MOOC rampant. Dist Educ 2013;34:253-63.
22. Baggaley J. Reflection: MOOC postscript. Dist Educ 2014;35:126-32. [Crossref]
23. Jordan K. Initial trends in enrolment and completion of massive open online courses. Int Rev Res Open Dist Learn 2014;15:133-59. [Crossref]
24. Melkun CH. Nontraditional students online: composition, collaboration, and community. J Cont High Educ 2012;60:33-9. [Crossref]
25. Toven-Lindsey B, Rhoads RA, Lozano JB. Virtually unlimited classrooms: Pedagogical practices in massive open online courses. Internet High Educ 2015;24:1-12. [Crossref]
26. Pantò E, Comas-Quinn A. The challenge of open education. J E-Learn Knowl Soc 2013;9:11-22.