Considering Learning and Evidence of Impact in Evaluating the Potential of AI for Education

Amy Corbitt — Tue, 29 Oct 2024 16:10:09 +0000

Considering Learning and Evidence of Impact in Evaluating the Potential of AI for Education Amy Corbitt Tue, 10/29/2024 - 10:10

William Penuel

William R. Penuel is a professor of learning sciences and human development in the School of Education at the University of Colorado Boulder. His current research examines conditions needed to implement rigorous, responsive, and equitable teaching practices in STEM education. At iSAT, he is a Co-Principal Investigator and Co-Lead of Strand 3 - which focuses on inclusive co-design processes to empower stakeholders with diverse identities to envision, co-create, critique, and apply AI learning technologies for their schools and communities.

As school and district leaders, you are used to building planes while flying them. But the advent of AI—specifically Generative AI—in classrooms has caught many of us off guard and not sure what airspace we’ve entered. Generative AI is the technology behind popular tools like ChatGPT, as well as tools today that use AI to help teachers build lesson plans and assessments for use in their classrooms. It’s a specific kind of AI that learns from the data it’s been fed (such as text, video, or images) to create new content. If you’ve tried it out, you may be impressed both by its capabilities to simulate human interaction, as well as its limitations.

As an education leader, Generative AI presents many interrelated challenges to you, to teachers, to parents, and to students pertaining to safety, transparency, and ethics. In this blog post, we want to focus on two other central issues that Chief Academic Officers, district technology leaders, principals, and instructional coaches should keep in the foreground when evaluating the potential integration of AI into schools: learning and evidence of impact. Learning has to do with both our goals for learning and how we support them. Evidence of impact has to do with the power and limits of tools to achieve those learning goals. Good evidence also involves evidence of what’s required of teachers to implement tools well, to achieve benefits for students. Both these considerations are important in evaluating Generative AI and other tools, but often they live in the background of discussions about Generative AI.

Take the discussion of the potential of Generative AI for personalization and differentiation of learning. This is chief among the advantages that advocates of AI tout. The questions to consider are: What kinds of learning goals can Generative AI support? What do we know about the potential of Generative AI for supporting these goals?

Intelligent Tutors Help Personalize Individuals’ Mastery of Discrete Knowledge and Skills

There is more than 50 years of research on intelligent tutoring systems (ITSs) that we can draw on to give us a sense of what learning goals AI for personalization can support. ITSs are trained when their developers subdivide knowledge to be taught into smaller components—skills, abilities, and concepts—allowing ITSs to recommend tasks based on a student’s mastery level. There’s a large body of evidence of impact that suggests that for the kinds of problems ITSs are used to help students with, they do as least as well as human tutors do in supporting learning.

However, while AI excels at guiding students toward specific, well-defined learning goals (like solving a math problem), it struggles with more open-ended tasks where multiple solutions exist, or where collaboration and dialogue are essential. Further, it may limit deeper engagement and valuable experiences like productive struggle or peer collaboration. The evidence base applies only to well-designed ITSs, as well. Many of the Generative AI tools today can’t achieve the results of the best ITSs. While they are good at handling requests in everyday language, many of these tools still give inaccurate answers to math problems students encounter in schools.

This is not to say that Generative AI won’t become more capable of solving math problems or helping support critical thinking, teamwork, and real-world problem solving in the future, but there is not strong evidence of impact for achieving these learning goals. There is even less evidence related to what’s needed to prepare teachers to use these tools well. There’s reason to be skeptical, then, about claims that the current class of tools of Generative AI can support these goals.

AI Can Support Collaborative Problem Solving in Inquiry-Rich Environments

There’s an equally rich body of evidence of impact for a set of AI tools that support collaborative learning. For more than two decades, the field of computer-supported collaborative learning has created and tested different tools focused on fostering group awareness and giving students feedback on small groups’ cognitive and social dynamics. A review of these kinds of group awareness tools show improvements to students’ knowledge and skill, as well as group task performance and social interaction in collaborative learning. The relevance of these findings for K-12 schools, though, is not as clear, because many of these tools were designed for online environments in higher education.

Here’s where emerging research comes in – the kind designed to build evidence of impact grounded in a robust vision for teaching and learning. The Institute of Student AI-Teaming is developing AI partners—the Community Builder (CoBi) and the Jigsaw Interactive Agent (JIA)—that perform the key functions of group awareness tools. These tools are intended to be integrated with rich curricula that focus on collaborative problem solving in STEM. These tools do something very different from what Generative AI tools as currently used to plan instruction or support personalization do: they help students learn to collaborate more effectively and equitably. They support a different kind of learning, too, one that is focused on students figuring out ideas and solving problems together, using disciplinary practices from STEM that are targeted in today’s standards. And while we are still gathering evidence of impact, we already know that students are using some collaborative solving skills more when they are using an AI partner to support their learning. We aim to make these partners—and the instructional materials to teach about AI—available to schools for free in the coming year.

Questions to Ask �� Learning and Impact

AI is here to stay, and as a leader, you know you have an obligation to approach how to use AI responsibly and ethically to achieve your vision for teaching and learning. No doubt, AI may now or in the future be useful for increasing efficiency in how teachers plan and how students develop discrete knowledge and skill. As vendors continue to rush to offer generative AI products to schools and districts, it’s important to ask three questions:

What kind of learning does this tool support?

What kind of preparation do teachers need to use the tool well?

What evidence of impact is there for the claims being made about Generative AI?

Integrating AI into classrooms is likely to lead to changes in how teachers teach and how students learn. Teachers will need support in learning how the AI works, and how to use AI tools to support teaching and learning that is consistent with what we know about how students learn. A generative AI chat bot doesn’t understand how people learn, no matter how skillful its interactions seem. That leaves it as your responsibility as a critical consumer of AI tools to ask tough questions of vendors about their ideas about teaching and learning and to demand they present evidence of bold claims about the power of AI.

Now is a moment when we are all particularly open and keen to learn about AI, and it is as imperative as ever to create opportunities where educators and leaders can learn together about the potential and limits of Generative AI and other tools that support learning goals for collaborative problem solving. We not only have to be “in the loop”: as decision makers about teaching and learning, we need to stay at the center, working at a pace that protects both our children and takes care of our visions for teaching and learning and that follows evidence more than hype.

Off

Traditional

White

Where Does the Data Go? A Behind-the-Scenes Look at iSAT’s Security Measures for Classroom Data Collection and Handling

Amy Corbitt — Fri, 18 Oct 2024 01:24:29 +0000

Where Does the Data Go? A Behind-the-Scenes Look at iSAT’s Security Measures for Classroom Data Collection and Handling Amy Corbitt Thu, 10/17/2024 - 19:24

By Charis Clevenger

With a Master's in Family and Human Development, Charis’s personal research interests include AI in education, relationship building, and learning through collaboration, equity in public schools, and viewing learning through the biopsychosocial model.

Do you ever wonder what happens to student data once the microphones and cameras are out of the classroom? With AI in education, there can be a lot of questions and concerns about how �� Boulder is protecting students’ information, whether it be their name, voice, image, or even the work they submit in class. It is challenging enough to navigate the school age years – worrying about how data remains secure shouldn’t be one of the contributing factors.

My name is Charis Clevenger, and I am the data manager for the Institute of Cognitive Sciences and iSAT. As a mother and former educator, the protection of vulnerable populations including our children is a critical motivating force in my role as data manager. Having been with iSAT since its founding (we are now in year 5), I make it a priority to ensure that we keep up to date with the latest best practices and safest measures for securing the data we collect.

iSAT, as a whole, is committed to following the Responsible Innovation Framework proposed by Stilgoe and colleagues (2013) where we protect the future from harm by emphasizing a stewardship of science and innovation in the present. Below are some ways how we apply this framework for our research policies on collecting data in classrooms.

Anonymizing personally identifying information at every stage

The first step after we collect data involves removing any information from the data that can identify a student participant. For this, we use study IDs instead of students’ real names. We also anonymize any information about their context, whether it’s who their teacher is, which school they attend, and what district they are in. Additional measures we take are:

Using untraceable identification numbers,
Blurring videos used for general analysis,
Transcribing speech to minimize the need for additional video use.

Ensuring raw data is secure once collected

Data is kept on secure servers that are password protected. Data collectors follow rigorous cyber security protocols and safeguards such as never “staying logged in” to any data networks.

Additionally, iSAT has put into place the careful curation of datasets based on specific needs from our in-house expert research teams. This happens only after the collected data has been rigorously checked and rechecked for any issue that could reveal identifying information. For example, suppose there is a school announcement made over the intercom during data collection and it may contain identifying information about the school; if this ends up being audible on the recording, we remove it. In doing so, our team ensures that collected data has to pass several levels of inspection and cleaning as well as move through various access control channels before it ever gets forwarded to research teams. And then we also track what data is being used and by whom. This minimizes the access to data that is not necessary to complete research by any given team.

In summary, it is imperative to update and refine security measures that protect the privacy of student participants. That is why iSAT has created a system that runs all collected data through various pre-processing and cleaning stages, limits access to data for research purposes only, and securely stores data for the lifetime of its use.

Off