Converse to Learn Project


PI: Mark Warschauer, University of California Irvine, School of Education

Co-PI: Andres Bustamante, University of California Irvine, School of Education

Sara Dewitt, PBS KIDS 

Abby Jenkins, PBS KIDS

Investigator: Ying Xu, University of California Irvine, School of Education

Joseph Aubele, University of California Irvine, School of Education

Funding Source: NSF Advancing Informal STEM Learning (AISL) Grant No. 1906321

Summary

Young children learn best – whether in daily life, reading books, or watching television – when they socially interact with an interested, caring, and knowledgeable language partner. In recent years, the rapid development of artificial intelligence (AI) has made conversational agents (CAs) more capable of simulating natural interpersonal interactions. Children’s natural reactions to CAs may lead to the inclusion of intelligent interfaces as language partners. CAs may be designed to converse with children while reading to them, providing children with ubiquitous learning opportunities. 

We are exploring two educational applications of conversational agents, including i) audio storybooks to promote early language and literacy skills and ii) interactive videos to foster scientific knowledge and curiosity.

Media Coverage

  • Ying Xu awarded Public Impact Fellowship for creating interactive videos to teach children science. UCI School of Education. November 25, 2019. [Link
  • UCI receives NSF grant to create and test interactive videos to foster children’s science learning. UCI News. September 17, 2019. [Link]

Project 1: CA for Early Language and Literacy Skills 

In this project, we have designed a CA-based audio story using Google’s voice-driven interface (Google Assistant). The CA pauses at particular points in the story and prompts children to answer an open-ended question. The CA gives feedback on the children’s responses, explaining why the answer is correct or incorrect. In cases where children fail to produce comprehensible answers (due to fuzzy pronunciation or a lack of comprehension of the prompt), the CA rephrases the original question in a multiple-choice format. 

A two-month-long field study was conducted to iteratively revise the conversational design of the agent. A two-by-two factor experiment (N = 120) was then carried out to examine the effectiveness of the revised agent, with the two factors being the narrator (the CA or a human researcher) and the inclusion of dialogue. Our results on learning outcomes indicated that 1) having conversations with a CA or a human partner significantly improved children’s reading comprehension; 2) having conversations with a CA resulted in similar comprehension levels as having conversations with a human partner.

We have also explored children’s perceptions of CAs as social others or informants, children’s verbal and non-verbal communications, visual attention, and emotion engagement when conversing with CAs.

Publications

  • Xu, Y., & Warschauer, M. (2020). What Are You Talking To?: Understanding Children’s Perceptions of Conversational Agents. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. April 25-30, 2020, Honolulu, HI. ACM. (Conditionally Accepted)
  • Xu, Y., & Warschauer, M. (2019). Young children’s reading and learning with conversational agents. In CHI Conference on Human Factors in Computing Systems Extended Abstracts. May 4–9, 2019, Glasgow, Scotland, UK. ACM. https://doi.org/10.1145/3290607.3299035

Conference Presentations

  • Xu, Y., Hoang, T., Sun, B, & Warschauer, M. (April 2020). What Are You Talking To?: Understanding Children’s Perceptions towards Conversational Agents. Paper to be presented at the Annual Meeting of the American Educational Research Association, San Francisco, CA.
  • Xu, Y., Lee, H., Bautista S., & Warschauer, M. (April 2020). Examining the Effect of a Conversational Agent as a Reading Partner. Paper to be presented at the Annual Meeting of the American Educational Research Association, San Francisco, CA.

Project 2: CA for Scientific Knowledge and Curiosity

In this project, we will develop interactive videos as a supplementary part of Elinor Wonders Why, a new PBS KIDS animated television program created by UCI Physics professor Daniel Whiteson and cartoonist Jorge Cham. The interactive videos allow children to directly speak with Elinor as she solves everyday science mysteries, thus priming children to engage in observation, prediction, pattern identification, and problem solving through scaffolded conversation. 

The project is guided by five foci:

  1. Can CAs feasibly help young children engage with and learn from science video watching?
  2. If so, how should the conversational experiences be best designed?
  3. What are the effects of using CAs to scaffold children’s video watching?
  4. Do these effects vary by children’s age, gender, English language proficiency, and family SES?
  5. How do parents and children perceive these interactive videos?

This project consists of three phases within two research cycles. In the development phase, we will develop the storyboard and conversation prompts and follow-ups, create animated videos based on the revised script (assisted by PBS KIDS Digital), and develop a mobile application of the interactive video integrated with the CA. In the field study phase, we will evaluate the initial product with 20 students. In the pilot RCT stage, we will conduct a four-way experiment with 40 children in each group (160 children in total): (1) watching Elinor Wonders Why with a CA (embodied in Elinor), (2) watching Elinor Wonders Why with a human partner, (3) watching Elinor Wonders Why with pseudo interactions where Elinor asks questions but does not listen or understand children’s answer, and (4) watching Elinor Wonders Why with no dialogue. 

Conference Presentations

  • Xu, Y., & Warschauer, M. (October 2019). Conversational Agents as Educational Video Co-viewers for Young Children. Paper presented at the 2019 Connected Learning Summit, Irvine, CA.