¶¶ÒõÂÃÐÐÉä

In this episode of Buff Innovator Insights, we meet Dr. Sidney D’Mello, Associate Professor at the Institute of Cognitive Science and the Department of Computer Science at ¶¶ÒõÂÃÐÐÉä Boulder. Dr. D’Mello describes how his unique combination of education and research experiences led to his leading-edge work at the NSF National Institute for Student-Agent Teaming, which is reimagining the role of artificial intelligence in the classroom to better serve both students and teachers. 

Terri Fiez

Hello, science fans, creative thinkers, and lifelong learners, and welcome to Buff Innovator Insights. I'm your host, Terri Fiez, Vice Chancellor for Research & Innovation at the University of Colorado Boulder. This podcast gives you a behind the curtain look at some of the most innovative and groundbreaking ideas in the world. Even better, it's an up-close and personal introduction to the people behind the innovations, from how they got their start to how they are making tomorrow better for all of us.

Today, I'll introduce you to Sidney D'Mello, Associate Professor at the Institute of Cognitive Science and the Department of Computer Science at ¶¶ÒõÂÃÐÐÉä Boulder. Before joining ¶¶ÒõÂÃÐÐÉä Boulder, he was an Associate Professor of Psychology and Computer Science at the University of Notre Dame. He has co-edited seven books and published more than 300 journal papers, book chapters and conference proceedings. D'Mello's research team is interested in the dynamic interplay between cognition and emotion while individuals and groups engage in complex real-world tasks. He now leads the NSF National AI Institute for student AI teaming, which aims to develop AI technologies to facilitate rich, collaborative learning experiences for all students.

In this episode, we'll learn about how Sidney's research and educational interest evolved through his unique college and graduate school experiences, the possibilities he has discovered while engaged in research and innovation at the intersection of multiple disciplines, and how his leading edge work at the Institute for Student AI Teaming is re-imagining the role of artificial intelligence in the classroom to better serve both students and teachers. Let's get started.

Hi Sidney. Thanks for joining me today.

Sidney D'Mello

Hi Terri, how are you?

Terri Fiez

Good. I'm really excited to talk about your background and the research that you do. Let's go ahead and get started. Growing up, what were some of your favorite subjects?

Sidney D'Mello

Realistically and honestly, I didn't really have a favorite subject. I just looked at school at the time as just something you do, but I really didn't quite get inspired by any real topic until later on. But generally I had kind of gravitation towards science and math.

Terri Fiez

Were there outside activities that you liked to pursue?

Sidney D'Mello

Yeah, very much so. I was a huge fan of sports, played a lot of soccer and running. And then later on I transitioned from playing more sports to more music. I really got into music pretty deeply. So actually I would say my main activities and pursuits at the time was really music and guitar and such.

Terri Fiez

Were there any people that influenced you in terms of your music that you were pursuing?

Sidney D'Mello

Yeah. I had a good set of friends and we tried to play together. One of my friends basically took classical guitar from a very young age, so he was very helpful in kind of showing me the ropes and always exploring new music. And that was a really formidable experience for me.

Terri Fiez

You have this interest in music and you sort of like math and science. How did you decide where to go to college?

Sidney D'Mello

So it really wasn't very much a large search or a big decision. I had friends who were going certain places and I decided to go to this small university in Memphis, Tennessee called Christian Brothers University. Among other things, it was really a financial thing. I didn't want to have large amounts of student loans. I knew some folks there and I thought Memphis... I was really big into the music at the time. So I think it just worked out, but it wasn't a very deliberate decision. It was just more, "Yeah, I'll go here," kind of thing.

Terri Fiez

And then how did you decide what to major in?

Sidney D'Mello

I had a view of schooling that was very extrinsically focused at the time. It was about you go to college, you get a degree, doing money. So I knew I wanted to do something that could get me a gig. So I decided to just pick engineering. I picked electrical engineering, electrical and computer engineering. This was about 1999, where there was a dot-com boom occurring at the time. So it just seemed like a really good choice in thinking about careers of the future.

Terri Fiez

Well, I think for many of us, it's always serendipitous how we end up where we do. It's helpful to hear that background.

Sidney D'Mello

Exactly.

Terri Fiez

So what were your experiences that stood out for you in your undergraduate program?

Sidney D'Mello

Two things come to mind. I took this wonderful class in world religion, and then we actually visited mosques and temples and churches and monasteries. That was really cool. My first actual class that really started getting me interested in academics in a more deeper way was we had this wonderful class on computer networks and it was very project oriented class where we basically spent at least two months focused on a project of real value. And that really made me think, "Wow, this is a whole different perspective on schooling than taking tests and doing homework."

Terri Fiez

So you graduated from college in 2002 when there was an economic downturn. How did that influence what you did after college?

Sidney D'Mello

I kind of did see the writing on the wall in terms of the jobs outlook around 2001. That's when I started thinking about going into graduate school, but realistically, I applied to a few graduate schools, but got rejected from almost all of them because I didn't know what I was doing. And rightly so. I knew nothing about what is a PhD? I didn't know what a journal paper was. So I then applied to a master's program at the University of Memphis in computer science, so just a local school. So I got admitted there and that's really, I think, another very serendipitous pathway that really got me into my PhD studies later on.

Terri Fiez

Very interesting. Well, I know in your master's degree, when we talked, you described how you kind of found your stride. Can you tell us about that?

Sidney D'Mello

Because it was actually the first time I got involved in actually doing research. I joined my graduate school advisor, professor Stan Franklin had a research group on AI and we were actually studying human consciousness, which is one of the most foundational questions to this date, probably second to where do we come from? And we focused on researching the topic in a very interesting way. It was a very interdisciplinary. It was my first deep exposure to psychology, to human cognition. And it was this blend of taking the best out of behavioral and cognitive sciences and then computing sciences. So building computer models of these really hard to pin down upon cognitive processes such as consciousness.

Terri Fiez

So the interdisciplinary part here was the combination of psychology and computer science. Is that right?

Sidney D'Mello

Yeah. And I would add to that a third one would be philosophy. Because a lot of these questions of consciousness and actually language that are really deep questions, having a philosophical lens to guide our thinking can help us kind of think through. Because there are limits. At the time, definitely there were limits to what you can get through the regular tools of the scientific practice, experiments and brain scans and computer models. But philosophy offered an opportunity to think about these questions and guide the science in a very different way.

Terri Fiez

So now, as you talk about this, it sounds very abstract, but actually your work was very grounded. Could you talk more about the work itself?

Sidney D'Mello

One of the projects I've started working on was called Intelligent Distribution Agent IDA. The basic idea was to help to understand the job of what they call dispatchers of the Navy that essentially matched sailors to trains and to ships. It's a very complicated process because I think there was 300,000 sailors at the time. Right? But what was interesting was we approached this and instead of just taking off the shelf tools from computer science, we try to actually emulate the cognitive processes of these dispatchers by studying them and actually emulating the decision-making. This is an example where we were taking a very real world practical problem, but using that as a context to really make scientific discoveries and technological discoveries. So that's an example of how we were trying to model cognition computationally, but in the context of an actual real world problem of value.

Terri Fiez

Now, I know you moved into research where you were looking at emotions, and the emotions project. Can you talk about that?

Sidney D'Mello

What occurred was there was another group actually across the hall, shall we say, that focused on developing intelligent tutoring. So it was the same idea. Like we were working with dispatchers, they worked with emulating human tutors. So human tutors are one of the most effective ways you can learn, one-on-one tutoring, but it's really expensive. It's really hard to scale. So the idea was to build computer programs that can actually emulate human tutors, but it turned out that a large part of what tutoring is and what underlies effective tutoring is that human tutors focus on students' emotions. They're attentive to students' motivational and emotional states. If a student is confused, they will offer a hand or if a student is frustrated, they will respond with some empathy and motivation and keep them going. This was completely missing from the computer tutors. So our project was actually focusing on bringing in some emotional intelligence to these computer tutors. And mind you, this was about 2003, 2004. This is when emotional intelligence was getting a lot of traction and getting into the mainstream.

Terri Fiez

The whole idea of EQ, emotional intelligence, was really taking off. And then you're looking at it in terms of how do we model that within a computer?

Sidney D'Mello

Yes, exactly. Exactly. So we first identified, what are the emotions? Then we focused on how can we measure them through humans? But then we said, "Okay, let's see if we can get computers to measure them." So we decided we wanted to target confusion, frustration, and boredom. And we looked at a multimodal approach. So we looked at analyzing facial expressions. This is when facial expression tracking, which is now a commercial product, was just getting off the road. We were working with some of the older cameras where you actually had to turn the lights off in the room so you can get reflections on the pupil and things like that. We looked at body movements. We had what we always lovingly referred to as the butt sensor. It essentially were pressure pads you put on the back in the seat of a chair and tracks your posture and your movement patterns. It's a very diagnostic signal. And we looked at the context. So looking at the context of the interaction with the tutor.

So with these three pieces of information, we were actually able to develop reasonably accurate models of confusion, frustration, and boredom. Then we said, "Okay, let's have these run live in the tutoring system so if it senses that you're confused, it would try to kind of motivate you." So we worked on what we call motivational scaffold. So if the tutor felt you were frustrated, it would respond with some encouragement and empathy. If it felt you were bored, it would try to liven things up. And if it was confused, it would try to reinforce the idea that actually confusion is not necessarily a negative thing and it's okay to be confused and we can help you work through this confusion. So the idea was this motivationally supportive tutoring system. And as a side note, we also contrasted that with the kind of humorous and kind of slightly snarky version that would kind of kid around. And actually some of the students really loved that one too.

Terri Fiez

Okay. So they responded to that. So clearly you did some really interesting work in your PhD. Where did you go from there?

Sidney D'Mello

I spent the next two and a half years, I stayed at the University of Memphis first as a postdoc for about a year. And then I switched to a research assistant professor, basically being able to kind of run a mini version of my own lab. And that's when we had a group of people, we advised students and so on and so forth. So that's kind of where I was. And then I had an opportunity through another project. I was recruited to Notre Dame where I guess they liked my interdisciplinary perspective and they actually created a position for me, which I'm always very grateful to them. That was actually half in computer science and half in psychology. So I was very much in these two departments that actually have different schools. So I moved there in 2012, spring.

Terri Fiez

So what research did you pursue at Notre Dame before coming to Boulder?

Sidney D'Mello

We were continuing our work in emotion. And then I realized that actually what we were noticing is that... I got really interested in the concept of boredom, which is another very understudied emotion. And we, at the same time, we got interested in a manifestation of boredom called mind wandering. And this is when you just zone out. And boy, isn't that tricky on how do you measure that? So we spent a lot of time actually understanding mind wandering in the context of reading, and then at the same time we developed mechanisms to detect it so automatically pick it out. And in this case, we started looking at eye movements, and then we also worked on interfaces that could kind of fully close the loop by sensing and responding to mind wandering.

So for one thing, you can actually make the reading interface more interactive. So we worked in a system. If it detects this mind wandering, it actually asked students to provide a self explanation. That brings their attention back. That helps them refocus, relearn, and we showed that had real benefits. So that's one way you can just help students on the fly.

But another thing is, frankly, a lot of times you can use it as a way to quantify the interesting ness of educational materials and certain topics. If students are all struggling on a certain topic or because of high mind wandering, the teacher can then decide to focus, "Okay, I'm going to focus on this topic." So it can help teachers plan their next classes in a way that's sensitive to basically students' attention.

One other thing I realize is that with the internet and social media and mobile phones, it became increasingly easy and important to do this research in the wild. So we started taking this whole program of research outside of the lab with all our control in the classrooms, into offices. And that really kind of expanded the bandwidth and I would like to say the significance and relevance of our work.

Terri Fiez

What prompted you to come to ¶¶ÒõÂÃÐÐÉä Boulder?

Sidney D'Mello

I really loved Notre Dame. I had a great set of colleagues there and I was there for five and a half years. I got tenured there. Again, it was just a matter of opportunity. I just randomly ran into professor Mike Mozer at a conference in DC at a meeting in DC. And he told me about this opportunity. And I was really excited. I wasn't even thinking of leaving, but it was the fact that I could be back in an Institute of Cognitive Science. That was just really exciting to me. I wanted to get into more neuroscience and the fact that we have a scanner here was important and relevant. And also, I wanted some sunshine and I also wanted to get healthy and get back into nature.

Terri Fiez

Great answer. Well, you were recently awarded one of the inaugural NSF AI, which stands for Artificial Intelligence Institutes. Can you tell us what the focus of that institute is?

Sidney D'Mello

So in our Institute, we're looking to see how we can re-imagine the role of AI in education. So previously with great success, AI has been used in with respect to personalized learning. And as I said, one-on-one tutoring and so on. So we said, how do we use AI and embed AI in classrooms? How can AI be part of the ecosystem between students, teachers, and small groups of students? So AI Institute is focusing on developing these AI partners. These are AI agents that work with small groups of students in very carefully designed curricula along with the teachers.

And the idea is for the students, the AI and the teachers to orchestrate meaningful and rich collaborative learning activities that are enhanced and supported by AI. It's like a teacher working with multiple AI partners together with students. That's kind of our vision. And underlying this vision really requires a lot of foundational AI challenges that we need to overcome such as simple as how do you do speech recognition in a noisy classroom environment? How do you do team science where now suddenly a member of the team is an artificial intelligent agent, as opposed to, for example, another student? And lastly, how do we address issues of equity and access and fairness in AI algorithms? How do we design AI with an ethics, equity, and justice focused mindset? So these are the challenges that our institute hopes to address by bringing a diverse set of folks together, all unified around a foundational mission.

Terri Fiez

And it also seems like one of the things that you're really helping to promote is additional leverage for our teachers so that they can be more effective in the classroom and be able to individualize the work and the support for students in the classroom.

Sidney D'Mello

What teachers are really good at and what they love to do is be with their students and help their students learn and grow. The more we can have AI help teachers achieve this by taking care of some of the more routine tasks and also enhancing the learning by working very closely with teachers is really where we'd love to be. And as a teacher myself, right? The time spent in the classroom is the best, most engaging, amazing time, but there's everything else around that. We can just focus teachers and allow teachers to have an opportunity to just do what they do best, which is love and nurture and help the students grow. Then we've been successful.

Terri Fiez

That's great. For you, getting involved in research really put a real focus on what your direction would be. How are students involved in your research and what do they get out of it?

Sidney D'Mello

I think what students get is multiple things, but the most important thing is for students to understand what is research and what is it like to pursue a career in research? And what is it like to do research? What is it like to have a question that only you are addressing and it's your question and there's no answer? I think those experiences can be quite transformative for many students. Some of them actually say, "Wow, I did this. This is not for me." And I think that's a great response and they want to do something different. We welcome diversity of experiences and opinions, but others say, "This is exactly what I want to do."

Terri Fiez

That's great. So what are you most proud of in your career so far?

Sidney D'Mello

If I have to think about what I'm most proud of, I'm really proud of the fact that I'm able to bring people together because I've had these different experiences and I've worked in all these different areas in my graduate training and in Notre Dame. I'm proud of my ability to kind of work at the intersection of different research areas in education, in emotion science, in cognitive science and in learning sciences and in computer science.

Terri Fiez

Yeah. And that will have huge impact bringing those disciplines together. So my final question is in the next decade or two, what are you optimistic about in your field and what is your hope about the future of the field?

Sidney D'Mello

So whether you know it or not or like it or not, AI is kind of embedded in our world, in our homes, in our communities, and in schooling. The question is where do we go from here? We have an opportunity to really redefine and at least attempt to redefine and reconsider how AI can be beneficial for folks. At the same time, we're very well aware that there are a lot of fears and concerns of AI with respect to surveillance, with respect to automation. And we also want to use this institute as an opportunity to really understand these concerns and work with communities to provide models of how AI in education can be implemented, that can stand up for other folks to use.

So for example, we've adopted a framework called the Responsible Innovation Framework at our institute where we ask certain questions every time we make decisions involving AI. And the central question really is what is it that we should do? Not just asking what AI can do. Because you can collect all kinds of data and you can do all kinds of things, but what are the responsible things to do? So we've also adopted this idea of co-design, of a way to design AI systems with the exact members of the community that they are impacting. So I hope that we will be in a place where maybe the conversation around AI in education has shifted. There's more accountability, but at the same time, there's actually more success and more impact.

Terri Fiez

Thank you, Sidney. This has been a very stimulating conversation. It's so exciting to see the kinds of work that you're doing and the impact that it's going to have over the years to come.

Sidney D'Mello

Thanks, Terri. It was my pleasure.

Terri Fiez

I hope you enjoyed my conversation with Sidney D'Mello, Associate Professor at the Institute of Cognitive Science and the Department of Computer Science at ¶¶ÒõÂÃÐÐÉä Boulder. You can learn more about Dr. D'Mello and the NSF National AI Institute for Student AI Teaming at . For more Buff Innovator Insights episodes and to join our email list visit colorado.edu/rio/podcast. I'm your host and Vice Chancellor for Research & Innovation at ¶¶ÒõÂÃÐÐÉä Boulder, Terri Fiez. It's been a pleasure to be with you. Innovation is for everyone. We can all make the world a more interesting and better place. Sometimes it just takes a spark. We'll see you next time.