From Keystrokes to Achievement Scores: The Moderating Effects of Computer Use on Writing
PIs:
Mark Warschauer, University of California, Irvine, School of Education
Jamal Abedi, University of California, Davis, School of Education
Investigators: Tamara Tate, University of California, Irvine, School of Education
Project Funding: Spencer Foundation, Grant No. 201500153, $49,882
Summary: Writing is difficult and changes in modality matter: Students perform better in whichever mode (e.g., pen and paper versus computer) they are most proficient (Horkay, Bennett, Allen, Kaplan, & Yan, 2006). Thus, whenever a singular mode of delivery is chosen for an assessment, some students are disadvantaged.
Concerns about the effect of computer-based assessments on students without regular access to technology abound. As educators implement computer-based assessments for NAEP and the Common Core State Standards, the impact of prior computer use becomes increasingly relevant to achievement scores.
This secondary data analysis looked at the process of digital writing as represented in the 2011 National Assessment of Educational Progress (NAEP). This assessment was the first large, national writing test given on computer to secondary students (see discussion in NAGB, 2010), allowing the collection of keystroke-level data to investigate the details of the process of digital writing. Students had access to common word-processing tools for editing (e.g., cut, copy, paste); formatting (e.g., indenting); spelling, grammar, and reference (spell check, thesaurus, dictionary); viewing and reviewing during the assessment (NAGB, 2010). In addition, the assessment included student and teacher questionnaires that gathered information about students’ prior use of computers. While descriptive relationships about these data have been published, inferential statistics have not been used to validate an accurate model of the impact of prior computer access and computer use during an assessment on writing achievement.
This research analyzed the NAEP data to determine the appropriate model of the impact of prior access to and use of computers (prior use) and use of the computer during the assessment (use during) on writing achievement. Current statistical methods allow us to model keystrokes and functions across groups of students, to understand the relationships between achievement and reported prior use of computers, and to deconstruct the differences between successful and less successful writers by gender, ethnicity, status as a current or former English language learner, socioeconomic status, and disability.
This study examined the following research questions:
1. What is the effect of prior computer use on writing achievement?
2. What is the effect of computer use during the assessment on writing achievement?
3. What is the relationship, if any, between prior computer use and computer use during the assessment in impacting achievement?
4. Are there heterogeneous effects across demographic groups?
We drew on a sociocultural perspective that views human activity as mediated by tools (Vygotsky, 1981; Wertsch, 1991). From this perspective, what is significant about various tools is not their abstract properties, but rather, when included in the process of behavior, how they may potentially alter the flow and structure of mental functions. Furthermore, this alteration unfolds developmentally, so that analysis of individual events needs to be paired with examination of the development of the individual (Vygotsky, 1962). This led us to query how the introduction of a powerful tool such as a computer may transform the writing process and how that transformation may be shaped by prior experiences in students’ lives.
Presentation:
American Educational Research Association (AERA) Annual Meeting 2015 Graduate Student Roundtable Presentation.
Mark Warschauer, University of California, Irvine, School of Education
Jamal Abedi, University of California, Davis, School of Education
Investigators: Tamara Tate, University of California, Irvine, School of Education
Project Funding: Spencer Foundation, Grant No. 201500153, $49,882
Summary: Writing is difficult and changes in modality matter: Students perform better in whichever mode (e.g., pen and paper versus computer) they are most proficient (Horkay, Bennett, Allen, Kaplan, & Yan, 2006). Thus, whenever a singular mode of delivery is chosen for an assessment, some students are disadvantaged.
Concerns about the effect of computer-based assessments on students without regular access to technology abound. As educators implement computer-based assessments for NAEP and the Common Core State Standards, the impact of prior computer use becomes increasingly relevant to achievement scores.
This secondary data analysis looked at the process of digital writing as represented in the 2011 National Assessment of Educational Progress (NAEP). This assessment was the first large, national writing test given on computer to secondary students (see discussion in NAGB, 2010), allowing the collection of keystroke-level data to investigate the details of the process of digital writing. Students had access to common word-processing tools for editing (e.g., cut, copy, paste); formatting (e.g., indenting); spelling, grammar, and reference (spell check, thesaurus, dictionary); viewing and reviewing during the assessment (NAGB, 2010). In addition, the assessment included student and teacher questionnaires that gathered information about students’ prior use of computers. While descriptive relationships about these data have been published, inferential statistics have not been used to validate an accurate model of the impact of prior computer access and computer use during an assessment on writing achievement.
This research analyzed the NAEP data to determine the appropriate model of the impact of prior access to and use of computers (prior use) and use of the computer during the assessment (use during) on writing achievement. Current statistical methods allow us to model keystrokes and functions across groups of students, to understand the relationships between achievement and reported prior use of computers, and to deconstruct the differences between successful and less successful writers by gender, ethnicity, status as a current or former English language learner, socioeconomic status, and disability.
This study examined the following research questions:
1. What is the effect of prior computer use on writing achievement?
2. What is the effect of computer use during the assessment on writing achievement?
3. What is the relationship, if any, between prior computer use and computer use during the assessment in impacting achievement?
4. Are there heterogeneous effects across demographic groups?
We drew on a sociocultural perspective that views human activity as mediated by tools (Vygotsky, 1981; Wertsch, 1991). From this perspective, what is significant about various tools is not their abstract properties, but rather, when included in the process of behavior, how they may potentially alter the flow and structure of mental functions. Furthermore, this alteration unfolds developmentally, so that analysis of individual events needs to be paired with examination of the development of the individual (Vygotsky, 1962). This led us to query how the introduction of a powerful tool such as a computer may transform the writing process and how that transformation may be shaped by prior experiences in students’ lives.
Presentation:
American Educational Research Association (AERA) Annual Meeting 2015 Graduate Student Roundtable Presentation.
