Converse to Learn Project

PI: Mark Warschauer (School of Education, University of California, Irvine)

Co-PI: 

Andres Bustamante (School of Education, University of California, Irvine)

Sara Dewitt, PBS KIDS 

Abby Jenkins, PBS KIDS

Investigators:

Ying Xu, University of California Irvine, School of Education

Valery Vigil, University of California Irvine, School of Education

Funding Source: NSF Advancing Informal STEM Learning (AISL) Grant No. 1906321

Summary

Young children learn best – whether in daily life, reading books, or watching television – when they socially interact with an interested, caring, and knowledgeable language partner. In recent years, the rapid development of artificial intelligence (AI) has made conversational agents (CAs) more capable of simulating natural interpersonal interactions. Children’s natural reactions to CAs may lead to the inclusion of intelligent interfaces as language partners. CAs may be designed to converse with children while reading to them, providing children with ubiquitous learning opportunities. 

We are exploring two educational applications of conversational agents, including i) audio storybooks to promote early language and literacy skills and ii) interactive videos to foster scientific knowledge and curiosity. 

This 11 minute presentation provides a nice summary of the project.

Media Coverage

  • Ying Xu awarded Public Impact Fellowship for creating interactive videos to teach children science. UCI School of Education. November 25, 2019. [Link
  • UCI receives NSF grant to create and test interactive videos to foster children’s science learning. UCI News. September 17, 2019. [Link]

Project 1: CA for Early Language and Literacy Skills 

In this project, we have designed a CA-based audio story using Google’s voice-driven interface (Google Assistant). The CA pauses at particular points in the story and prompts children to answer an open-ended question. The CA gives feedback on the children’s responses, explaining why the answer is correct or incorrect. In cases where children fail to produce comprehensible answers (due to fuzzy pronunciation or a lack of comprehension of the prompt), the CA rephrases the original question in a multiple-choice format.

A two-month-long field study was conducted to iteratively revise the conversational design of the agent. A two-by-two factor experiment (N = 120) was then carried out to examine the effectiveness of the revised agent, with the two factors being the narrator (the CA or a human researcher) and the inclusion of dialogue. Our results on learning outcomes indicated that 1) having conversations with a CA or a human partner significantly improved children’s reading comprehension; 2) having conversations with a CA resulted in similar comprehension levels as having conversations with a human partner.

We have also explored children’s perceptions of CAs as social others or informants, children’s verbal and non-verbal communications, visual attention, and emotion engagement when conversing with CAs.

Publications

  • Xu, Y., Wang, D., Collins, P., Lee, H., & Warschauer, M. (2020). Same benefits, different communication patterns: Comparing children’s reading with a conversational agent vs. a human partner. Computers & Education. https://doi.org/10.1016/j.compedu.2020.104059
  • Xu, Y., & Warschauer, M. (2020). A content analysis of voice-based apps on the market for early literacy development. In Proceedings of the 19th ACM International Conference on Interaction Design and Children (IDC ’20). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3392063.3394418 *Best Paper Award Honorable Mention
  • Xu, Y., & Warschauer, M. (2020). Exploring young children’s engagement in joint reading with a conversational agent. In Proceedings of the 19th ACM International Conference on Interaction Design and Children (IDC ’20). Association for Computing Machinery, New York, NY, USA. https://doi.org/ 10.1145/3392063.3394417
  • Xu, Y., & Warschauer, M. (2020). What are you talking to?: Understanding children’s perceptions of conversational agents. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. April 25-30, 2020, Honolulu, HI. ACM. https://doi.org/10.1145/3313831.3376416
  • Xu, Y., & Warschauer, M. (2019). Young children’s reading and learning with conversational agents. In CHI Conference on Human Factors in Computing Systems Extended Abstracts. May 4–9, 2019, Glasgow, Scotland, UK. ACM. https://doi.org/10.1145/3290607.3299035

Conference Presentations

  • Song, Y.*, Deng, X.*, & Xu, Y. (April 2021). Young children’s reading with conversational agents: The role of age and language status. Paper to be presented at the Annual Meeting of the American Educational Research Association, Virtual Conference
  • Warschauer, M., & Xu, Y. (September 2020). Can conversational agents support children’s learning?. Conference on Educational Data Science, Stanford, CA (Virtual conference).
  • Xu, Y., Hoang, T.*, Sun, B.*, & Warschauer, M. (April 2020). What Are You Talking To?: Understanding Children’s Perceptions towards Conversational Agents. Paper accepted at the Annual Meeting of the American Educational Research Association, Virtual Conference.Conference cancelled
  • Xu, Y., Lee,  H.*, Bautista S.*, & Warschauer, M. (April 2020). Examining the Effect of a Conversational Agent as a Reading Partner. Paper accepted at the Annual Meeting of the American Educational Research Association. Conference cancelled.
* Undergraduate research assistants in this project

Project 2: CA for Scientific Knowledge and Curiosity

In this project, we will develop “conversational videos”  as a supplementary part of Elinor Wonders Why, a new PBS KIDS animated television program created by UCI Physics professor Daniel Whiteson and cartoonist Jorge Cham. The conversational videos allow children to directly speak with Elinor as she solves everyday science mysteries, thus priming children to engage in observation, prediction, pattern identification, and problem solving through scaffolded conversation. 

Click here for a one minute video demonstration.
Elinor, the main character of Elinor Wonders Why. The conversational agent will be embodied in Elinor. ©2020 SHOE Ink, LLC

The project is guided by five foci:

  1. Can CAs feasibly help young children engage with and learn from science video watching?

  2. If so, how should the conversational experiences be best designed?

  3. What are the effects of using CAs to scaffold children’s video watching?

  4. Do these effects vary by children’s age, gender, English language proficiency, and family SES?

  5. How do parents and children perceive these interactive videos?

This project consists of three phases within two research cycles. In the development phase, we will develop the storyboard and conversation prompts and follow-ups, create animated videos based on the revised script (assisted by PBS KIDS Digital), and develop a mobile application of the conversational video. In the field study phase, we will evaluate the initial product with 20 children. In the pilot RCT stage, we will conduct a four-way experiment with 40 children in each group (160 children in total): (1) watching Elinor Wonders Why conversational video (talking with the agent embodied in Elinor), (2) watching Elinor Wonders Why with a human partner, (3) watching Elinor Wonders Why with pseudo interactions where Elinor asks questions but does not listen or understand children’s answer, and (4) watching Elinor Wonders Why with no dialogue in traditional, non-interactive format. Most of our participants are from Santa Ana, a working class, predominantly Latino community. 

We have completed the first research cycle, and here are what we have learned: 

  • The conversational video resulted in children’s better understanding of the science concepts. This positive effect was robust across children from both low and high income neighborhoods.

  • The conversational video had a compensatory effect for language minority children. 

  • The conversational video helped children form a positive relationship with the main character. 

  • The conversational video elicited children’s active verbal responses during video watching.

  • The conversational video also appeared to encourage parents’ active co-viewing. 

 

Our research so far has established the feasibility of integrating conversational agents in children’s STEM television programs. We will continue to partner with PBS KIDS to further the impacts of this innovative work by producing additional conversational videos based on other animation series and distributing the conversational videos as publicly accessible content at no cost to children and families through the PBS KIDS app and website.

Publications

  • Xu, Y., & Warschauer, M. (2020). Wonder with Elinor: Designing a socially contingent video viewing experience. In Proceedings of the 19th ACM International Conference on Interaction Design and Children (IDC ’20). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3397617.3398024 *Research and Design Competition Award Honorable Mention
  • Xu, Y., & Warschauer, M. (2020). “Elinor is talking to me on the screen!” Integrating conversational agents into children’s television programming. In CHI Conference on Human Factors in Computing Systems Extended Abstracts. April 25-30, 2020, Honolulu, HI. ACM. https://doi.org/10.1145/3334480.3383000

Conference Presentations

  • Xu, Y., & Warschauer, M. (October 2019). Conversational Agents as Educational Video Co-viewers for Young Children. Paper presented at the 2019 Connected Learning Summit, Irvine, CA.

UROP Fellows

The Converse to Learn project provides an excellent opportunity for undergraduate students to gain research experiences. Below are the list of students in this project awarded the UCI’s Undergraduate Research Opportunity Fellowships:

  • So Eun Lee, Biological Sciences
  • Emily Nguy, Education Sciences
  • Somin Joo, Psychological Sciences
  • Hyelim Lee, Psychological Sciences
  • Yongjia Song, Psychological Sciences
  • Branda Sun, Psychological Sciences and Education Sciences
  • Xinwei Deng, Psychological Sciences
  • Abigail Sarah Bautista, Psychological Sciences
  • Tran Hoang, Psychological Sciences and Education Sciences