Investigating Virtual Learning Environments

PI’s 
Mark Warschauer, University of California, Irvine, School of Education
Di Xu, University of California, Irvine, School of Education
Padhraic Smyth, University of California, Irvine, UCI Donald Bren School of Information and Computer Sciences
Teomara Rutherford, University fo Delaware, School of Education
Rachel Baker, University of California, Irvine, School of Education
Brian Sato, University of California, Irvine, School of Biological Sciences

Postdoctoral Scholar
Fernando Rodriguez, University of California, Irvine, School of Education
Christian Fischer, University of California, Irvine, Teaching and Learning Research Center

Research Staff
Kameryn Denaro, University of California, Irvine, School of Education
Tamara Tate, University of California, Irvine, Teaching and Learning Research Center

Graduate Students
Dennis Dang (Education)
Joseph Aubele (Education)
Yiwen Lin (Education)
Mariela Rivas (Education)
Renzhe Yu (Education)

Affiliates
Nia Dowell, Assistant Professor, University of California, Irvine
Qiujie Li, Post-Doc, New York University
Peter McPartlan, Post-Doc, San Diego State University

Funding Source: NSF Grant Number 1535300, 2015-2020

Online Learning Research Center: olrc.us

Summary

With 69% of higher education institutions stating that online learning is a critical part of their long-term strategy and 32% of higher education students taking at least one course online (Allen & Seaman, 2013), new forms of virtual learning are raising the specter of radically transformed undergraduate education. This is particularly true for lower-division courses in science, technology, engineering, and mathematics (STEM), which disproportionately rely on large lectures. The possibilities of providing lecture material online, as is done through the popular Khan Academy and new massive online open courses (MOOCs), suggest to many that virtual learning may have the potential to serve as a comparable or even advantageous alternative to traditional in person lectures. Debates on this issue highlight both the high hopes and potential pitfalls. To date, few rigorous studies have convincingly demonstrated the effects of virtual learning. There have been a number of studies, including some meta-analyses, that have found benefits linked with virtual learning; however, only a handful have used rigorous experimental or quasi-experimental methods to identify causal effects (e.g., Figlio, Rush, & Yin, 2010; Xu & Jaggars, 2013).

In sum, despite their increasingly important role and fast growth in the national education landscape, the field has limited information about the impacts of different course delivery formats on student learning outcomes and on potential strategies to improve their effectiveness. To fill this research gap, the project intends to systematically explore the effectiveness of virtual learning in the particular setting of STEM courses in large research universities. This setting offers the means to test moderation hypotheses regarding student populations, course content, and instructional practices within a context that plays a special role in the STEM pipeline to graduate study and advanced careers. In keeping with the theory of affordances (see Warschauer, 2003) in which we set our work, by conducting both experiments and observations across a broad swath of courses, we aim to flesh out the heterogeneous effects of virtual learning environments and identify what kinds of environments and instructional techniques are most potent for aiding diverse learners.

This project thus has three major goals:

1) to explore how different types of virtual learning environments affect teaching and learning processes, outcomes, and attitudes toward STEM, both in general and for particular groups of underrepresented learners

2) to examine how variation in instructional practices is associated with student learning gains for the purposes of distilling potential strategies to design high-quality virtual learning environments and improve student learning;

3) to understand the student experience of learning within virtual learning environments through investigation of learning behaviors and the ways in which these behaviors relate to student outcomes

For more information contact Mark Warschauer at markw@uci.edu.

References 

Arbaugh, J. B. (2001). How instructor immediacy behaviors affect student satisfaction and learning in web-based courses. Business Communication Quarterly, 64(4), 42-54.

Baker, R., Evans, B., Li, Q., & Cung, B. (2018). Does inducing students to schedule lecture watching in online classes improve their academic performance? An experimental analysis of a time management intervention. Research in Higher Education.

Bandura, A. (2006). Guide for constructing self-efficacy scales. In Self-efficacy beliefs of adolescents.

Bambara, C. S., Harbour, C. P., Davies, T. G., & Athey, S. (2009). Delicate engagement: The lived experience of community college students enrolled in high-risk online courses. Community College Review, 36(3), 219-238. 

Bernacki, M. L., Chavez, M. M., & Uesbeck, P. M. (2020). Predicting achievement and providing support before STEM majors begin to fail. Computers & Education, 158, 103999.

Bernacki, M.L., Vosicka, L., & Utz, J.C. (2019). Can a brief, digital skill training intervention help undergraduates “learn to learn” and improve their STEM Achievement?” Journal of Educational Psychology. http://dx.doi.org/10.1037/edu0000405

Bettinger, E., Doss, C., Loeb, S., Rogers, A., & Taylor, E. (2017). The effects of class size in online college courses: Experimental evidence. Economics of Education Review, 58, 68-85.

Brown, M., DeMonbrun, R.M., & Teasley, S (2017) LAK ’17, March 13 – 17, 2017, Vancouver, BC, Canada ACM DOI: http://dx.doi.org/10.1145/3027385.3027393

Scott E. Carrell Michal Kurlaender Working Paper 27312 http://www.nber.org/papers/w27312

Chen, P., Chavez, O., Ong, D. C., & Gunderson, B. (2017). Strategic resource use for learning: A self-administered intervention that guides self-reflection on effective resource use enhances academic performance. Psychological Science, 28(6), 774-785. 

Corbeil, J. R. (2003). Online technologies, self-efficacy, self-directed learning readiness, and locus of control of learners in a graduate-level web-based distance education program. Unpublished doctoral dissertation. University of Houston.

Cox, R. D. (2009). The college fear factor: How students and professors misunderstand one another. Cambridge, MA: Harvard University Press.

Cung, B. (2018). Personality and scheduling in online courses. Unpublished dissertation, University of California, Irvine.

Cung, B., Xu, D., & Eichhorn, S. (2018). Increasing Interpersonal Interactions in an Online Course: Does Increased Instructor Email Activity and Voluntary Meeting Time in a Physical Classroom Facilitate Student Learning? Online Learning, 22(3), 193-215.

Dabbagh, N., Bass, R., Bishop, M. J., Picciano, A. G., Sparrow, J., Costelloe, S., … & Wilson, S. J. (2019). Using Technology to Support Postsecondary Student Learning: A Practice Guide for College and University Administrators, Advisors, and Faculty. WWC 20090001. What Works Clearinghouse.

Department of Education (2018).  Office of Civil Rights Data.  Retrieved online July 22, 2018 from https://ocrdata.ed.gov/Home

Dimitras, A. L., Slowinski, R., Susmaga, R., & Zopounidis, C. (1999). Business failure prediction using rough sets, Eur. J. Oper. Res. 114(2), 263-280.

Ellenberg, J. H. (1996). Intent-to-treat analysis versus as-treated analysis. Drug Information Journal, 30(2), 535-544.

Ehrman, M. (1990). Psychology: Psychological factors and distance education. American Journal of Distance Education, 4(1), 10-24.

Eisenberg, E., & Dowsett, T. (1990). Student drop‐out from a distance education project course: A new method of analysis. Distance Education, 11(2), 231-253.

Fischer, C., McPartlan, P., Orona, G., Yu, R., Xu, D., & Warschauer, M. (2020)

Friesen, N., & Kuskis, A. (2013). Modes of interaction. Handbook of Distance Education, (Ed. M. Moore), 351-371. Routledge: NY.

Guglielmino, L. M., & Guglielmino, P. J. (2003). Identifying learners who are ready for e-learning and supporting their success. Preparing learners for e-learning, 18-33.

Hart, C. M. D., Friedmann, E. A. Z., & Hill, M. (2018). Online course-taking and student outcomes in California Community Colleges. Education Finance and Policy, 13, 42-71.

Heckman, J. J., & Pinto, R. (2015). Econometric mediation analyses: Identifying the sources of treatment effects from experimentally estimated production technologies with unmeasured and mismeasured inputs. Econometric Reviews, 34(1-2), 6-31.

Jaggars, S., & Xu, D. (April 2013). Predicting online student outcomes from a measure of course quality. Community College Research Center:  CCRC Working Paper No. 57.

Jaggars, S. S., & Xu, D. (2016). How do online course design features influence student performance? Computers & Education, 95, 270-284.

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An Introduction to Statistical Learning. New York: Springer.Kearsley, G. (1995). The nature and values of interaction in distance education. In Third Distance Education Research Symposium. University Park, PA: American Center for the Study of Distance Education.

Koester, McKay, Kedharnath, Finzell, Derry, Hayward, Shultz, Thanhouser, Dowell, Perrett (2019)

McDaniel, M. A., & Einstein, G. O. (2020). Training learning strategies to promote self-regulation and transfer: the knowledge, belief, commitment, and

planning framework. Perspectives on Psychological Science, 1745691620920723.

Morehead, K., Rhodes, M. G., & DeLozier, S. (2016). Instructor and student knowledge of study strategies. Memory, 24(2), 257-271.

National Center for Public Policy and Higher Education. (2011). Affordability and transfer: Critical to increasing baccalaureate degree completion (Policy Alert). San Jose, CA

Rodriguez, F., Rivas, M. J., Matsumura, L., Warshauer, M., & Sato, B. (2018a). How do students study in STEM courses? Findings from a light touch intervention and its relevance for underrepresented students. PLoS ONE. 13(7), 1-20.

Rodriguez, F., Kataoka, S., Rivas, M., Kandandale, P., Nili, A., & Warschauer, M. (2018b). Do spacing and self-testing predict learning? Active Learning in Higher Education, 1-15.

Roll, I., & Winne, P. H. (2015). Understanding, evaluating, and supporting self-regulated learning using learning analytics. Journal of Learning Analytics, 2(1), 7-12.

Tullis, J. G. (n.d.).  E-Learning:  The opportunities and challenges of online instruction. University of Arizona. Retrieved November 4, 2020 from http://u.arizona.edu/~tullis/pubs/Tullis2020.pdf

Uzuntiryaki, E., & Aydin, Y. Ç. (2009). Development and validation of chemistry self-efficacy scale for college students. Research in Science Education, 39(4), 539–551.

Xu, D., & Jaggars, S. S. (2011). The effectiveness of distance education across Virginia’s community colleges: Evidence from introductory college-level math and English courses. Educational Evaluation and Policy Analysis, 33(3), 360-377.\

Xu, D., & Jaggars, S. S. (2014). Performance gaps between online and face-to-face courses: Differences across types of students and academic subject areas. The Journal of Higher Education, 85(5), 633-659.

Xu, D., Li, Q., & Zhou, X. (2020). Online Course Quality Rubric: A Tool Box. Online Learning Research Center, University of California, Irvine.

Xu, D., & Xu, Y. (March 2019).  The Promises and Limits of Online Higher Education: Understanding How Distance, Education Affects Access, Cost, and Quality.  American Enterprise Institute.

Zhou, N., Fischer, C., Rodriguez, F., Warschauer, M, & King, S. (2019). Exploring how enrolling in an online organic chemistry preparation course relates to students’ self-efficacy. Journal of Computing in Higher Education.

Zusho, A., Pintrich, P. R., & Coppola, B. (2003). Skill and will: The role of motivation and cognition in the learning of college chemistry. International Journal of Science Education, 25(9), 1081–1094.

Selected publications and presentations relating to prior NSF grant:

Investigating Virtual Learning Environments (NSF DUE: 1535300; Warschauer, PI, Xu & Baker Co-PIs)

Baker, R., Evans, B., Li, Q., & Cung, B. (2019). Does inducing students to schedule lecture watching in online classes improve their academic performance? An experimental analysis of a time management intervention. Research in Higher Education, 60(4), 521-552.

Baker, R., Xu, D., Park, J., Yu, R., Li, Q., Cung, B., … & Smyth, P. (2020). The benefits and caveats of using clickstream data to understand student self-regulatory behaviors: opening the black box of learning processes. International Journal of Educational Technology in Higher Education, 17, 1-24.

Casasola, T., Nguyen, T., Warschauer, M., & Schenke, K. (2017). Can flipping the classroom work? Evidence from undergraduate Chemistry. The International Journal of Teaching and Learning in Higher Education, 29(3), 421-435.

Choi, H., Dowell, N. M., Brooks, C., & Teasley, S. D. (2019). Social comparison in MOOCs: Perceived SES, opinion, and message formality. In Proceedings of the 9th International Conference for Learning Analytics & Knowledge (pp. 160-169).

Cung, B., Xu, D., Eichhorn, S., & Warschauer, M. (2019). Getting academically underprepared students ready through college developmental education: Does the course delivery format matter? American Journal of Distance Education.

Cung, B., Xu, D., & Eichhorn, S. (2018). Increasing interpersonal interactions in an online course: Does increased instructor e-mail activity and a voluntary in-person meeting time facilitate student learning? Online Learning Journal, 1-17. doi 10.1080/08923647.2019.1582404

Fischer, C., Baker, R., Li, Q., Orona, G., & Warschauer, M. (2019, April). Does course-taking increase distal student success? Examining impacts on college graduation rates and time-to-degree. Paper presented at the annual meeting of the American Educational Research Association (AERA), Toronto, Canada.

Fischer, C., Baker, R., Li, Q., Rodriguez, F., Xu, D., & Warschauer, M. (2018, October). Online learning in higher education: Examining short-term and distal outcomes. Poster presented at the CRESST Conference, Los Angeles, CA.

Fischer, C., McPartlan, P., Orona, G., Yu, R., Xu, D., & Warschauer, M. (2020). Salient syllabi: Examining design characteristics of online courses in higher education. (Working paper)

Fischer, C., Baker, R., Li, Q., Orona, G., & Warschauer, M. (2019, April). Does course-taking increase distal student success? Examining impacts on college graduation rates and time-to-degree. Paper presented at the 2019 annual meeting of the American Educational Research Association, Toronto, Canada.

Fischer, C., Xu, D., Rodriguez, F., Denaro, K., & Warschauer, M. (2020). Effects of course modality in summer session: Enrollment patterns and student performance in face-to-face and online classes. The Internet and Higher Education, 45, 1-9.

Fischer, C., Baker, R., Li, Q., Rodriguez, F., Xu, D., & Warschauer, M. (2018, October). Online learning in higher education: Examining short-term and distal outcomes. Poster presented at the 2018 CRESST Conference, Los Angeles, CA.

Fischer, C., Pardos, Z. A., Baker, R. S., Williams, J. J., Smyth, P., Yu, R., … & Warschauer, M. (2020). Mining big data in education: Affordances and challenges. Review of Research in Education, 44(1), 130-160.

Fischer, C., Xu, D., Rodriguez, F., Denaro, K., & Warschauer, M. (2020). Effects of course modality in summer session: Enrollment patterns and student performance in face-to-face and online classes. The Internet and Higher Education, 45, 100710.

Fischer, C., Zhou, N., Rodriguez, F., Warschauer, M., & King, S. (2019). Improving College Student Success in Organic Chemistry: Impact of an Online Preparatory Course. Journal of Chemical Education, 96(5), 857-864.

Glick, D., Cohen, A., Festinger, E., Xu, D., Li, Q., & Warschauer, M. (2019). Predicting Success, Preventing Failure. In Utilizing Learning Analytics to Support Study Success (pp. 249-273). Springer, Cham.

He, W., Holton, A., Gu, H., Warschauer, M., & Farkas, G. (2019). Differentiated impact of flipped instruction: When would flipped instruction work or falter?. International Journal of Teaching and Learning in Higher Education, 31(1), 32-49.

He, W., Holton, A., Farkas, G., & Warschauer, M. (2016). The effects of flipped instruction on out-of-class study time, exam performance, and student perceptions. Learning and Instruction, 45, 61-71.

Jiang, S., Schenke, K., Xu, D., Eccles, J. S., Warschauer, M. (2018). Cross-national comparison of gender differences in the enrollment in and completion of science, technology, engineering, and mathematics Massive Open Online Courses. PLOS One, 13(9): e0202463.

King, S., Zhou, N., Fischer, C., Rodriguez, F., & Warschauer, M. (2019). Enhancing student learning and retention in organic chemistry: Benefits of an online organic chemistry preparatory course. In S. Kradtap Hartwell & T. Gupta (Eds.), From General to Organic Chemistry: Courses and Curricula to Enhance Student Retention (pp. 119-128). Washington, DC: American Chemical Association.

Li, Q., & Baker, R. (2018). The different relationships between engagement and outcomes across participant subgroups in Massive Open Online Courses. Computers & Education.

Li, Q., Baker, R., & Warschauer, M. (2018). Measuring student self-regulated learning in an online class. In Proceedings of the Eight International Learning Analytics & Knowledge Conference

Li, Q., Baker, R., & Warschauer, M. (2020). Using clickstream data to measure, understand, and support self-regulated learning in online courses. The Internet and Higher Education, 45, 100727.

Li, Q., Zhou, X., Bostian, B., & Xu, D. How Can We Improve Online Learning at Community Colleges?: Voices from Online Instructors and Students.

McPartlan, P., & Rutherford, T. (2018, April). Are our measures offline? Critiquing measures of motivation in online courses. Poster presented at the annual meeting of the American Educational Research Association, New York, NY.

McPartlan, P., Rutherford, T., Rodriguez, F., Shaffer, J. (2017, August). Modality motivation: Assessing motivational differences in online and face-to-face students. Paper presented at the annual meeting of the American Psychological Association Conference. Washington, DC. (Best Graduate Student Poster)

McPartlan, P., Li, Q., Rutherford, T., Yu, R., & Xu, D. (2020). Challenges of Improving Peer Interaction in Online Courses: The Costs of Social Presence. Accepted at the annual meeting of the American Educational Research Association (AERA). (Cancelled due to COVID-19)

McPartlan, P., Li, Q., Umarji, O., & Rutherford, T. (2019, May). How students with performance goals compare themselves when class is online. Paper presented at the annual meeting of the Association for Psychological Science, Washington, D.C.

Park, J., Yu, R., Rodriguez, F., Baker, R., Smyth, P., & Warschauer, M. (2018). Understanding student procrastination via mixture models. In Proceedings of the 11th International Conference on Educational Data Mining. Buffalo, NY.  Best Paper Award. 

Park, J., Denaro, K., Rodriguez, F., Smyth, P., & Warschauer, M. (2017). Detecting changes in student behavior from clickstream data. In Proceedings of the seventh International Learning Analytics and Knowledge Conference. Vancouver, BC, Canada. Best Paper: Honorable Mention.

Rodriguez, F., Rivas, M. J., Matsumura, L., Warshauer, M., & Sato, B. (2018a). How do students study in STEM courses? Findings from a light touch intervention and its relevance for underrepresented students. PLoS ONE. 13(7), 1-20.

Rodriguez, F., Kataoka, S., Rivas, M., Kadandale, P., Nili, A., & Warschauer, M. (2018b). Do spacing and self-testing predict learning? Active Learning in Higher Education, 1-15.

Rodriguez, F., Yu, R., Park, J., Rivas, M. J., Warschauer, M., & Sato, B. K. (2019, March). Utilizing Learning Analytics to Map Students’ Self-Reported Study Strategies to Click Behaviors in STEM Courses. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge (pp. 456-460).

Umarji, O., McPartlan, P., Li, Q., & Eccles, J. (2019). The carrot or the stick: The role of regret, satisfaction, and motivation in pursuit of daily academic goals in an online course. Paper presented at the annual meeting of the American Educational Research Association.

Wu, L. L., Fischer, C., Rodriguez, F., & Washington, G. (2018). Evaluation of online learning in a first-year engineering design course. Proceedings of the 2018 annual conference and exposition of the American Society of Engineering. Education, Salt Lake City, UT.

Xu, D., Glick, D., Rodriguez, F., Cung, B., Li, Q., & Warschauer, M. (2019). Does blended instruction enhance English language learning in developing countries? Evidence from Mexico. British Journal of Educational Technology.

Xu, D., Ran, X., & Zhou, X. (2019). Adopting online learning in college developmental education coursework: Impact on student course persistence, completion, and subsequent success. In Center for the Analysis of Postsecondary Readiness Conference. New York. US.

Xu, D., & Xu, Y. (2019).  The promises and limits of online higher education:  Understanding how distance education affects access, cost, and quality.  American Enterprise Institute.  https://www.aei.org/wp-content/uploads/2019/03/The-Promises-and-Limits-of-Online-Higher-Education.pdf

Yu, R. (2019). Deconstructing the Evolution of Collaborative Learning Networks. In Companion Proceedings of the 9th International Conference on Learning Analytics & Knowledge (LAK ’19). (pp. 741-745).

Yu, R., Li, Q., Fischer, C., Xu, D., & Doroudi, S. Predicting College Success: What Data Are Useful and for Whom? (2020). In Companion Proceedings of the 10th International Conference on Learning Analytics & Knowledge (LAK ’20).

Yu, R., Jiang, D., & Warschauer, M. (2018). Representing and Predicting Student Navigational Pathways in Online College Courses. In Proceedings of the 5th ACM Conference on Learning at Scale. London, United Kingdom.

Yu, R., Pardos, Z., & Scott, J. (2019). Student Behavioral Embeddings and Their Relationship to Outcomes in a Collaborative Online Course. In Learning Analytics: Building Bridges Between the Education and the Computing Communities Workshop at the 12th International Conference on Educational Data Mining (EDM ’19). Montreal, QC, Canada.

Yu, R. & Xu, D. What Do You Get from Replies? Causal Estimates of Peer Effects in Online Discussion Forums. (2020). In Companion Proceedings of the 10th International Conference on Learning Analytics & Knowledge (LAK ’20).

Zhou, N., Fischer, C., Rodriguez, F., Warschauer, M, & King, S. (2019). Exploring how enrolling in an online organic chemistry preparation course relates to students’ self-efficacy. Journal of Computing in Higher Education.

Zhou, X., Li., Q., & Xu, D. What Makes a Successful Online Learner?:  Community College Students’ Perceptions of Online Learning Strategies. (Working paper)

 

Training and Software Products

Intro to Python for Data Science

Python for Clickstream Analysis

Canvas API Crawler for Accessing Learning Analytic Data