My Experiences with Khan Academy and IRT in the Classroom

First of all, I absolutely loved the readings, videos, and assignments for this week’s T509 class.  All of the content was all around really awesome stuff and full of fascinating ideas.

I could not pass up the opportunity to write about some of my experiences with Kahn Academy.  I am a huge fan of their system and have used it in my math classroom for the past three years.  I also tend to find myself spending an hour or two here or there practicing my own math skills as well (kind of really proud of my Profile).

To establish some context, I started out my career teaching a 9th grade math intervention course.  I was plopped in a computer lab with absolutely zero curriculum tools and given the most behind 9th graders and told to do something about it.  After some testing, I found that my students were on average at a 5th grade math level and needed a lot of support if they were to be successful in Algebra 1.  Kahn Academy was one of the first places I went to to find solutions to this problem.  Back then, the interface looked something like this:

Students could navigate the “knowledge map” at their own pace.  Blue squares represented mastered skills, green recommended, and orange was a previously mastered skill that the system believes needs to be reviewed.  From my understanding, this system was not grounded in IRT at all.  It simply displayed a curriculum map under the assumption that completing one skill meant that the student was ready to move on to the next and if they struggled with that next skill, then they should go back and review the skill that came before it.

As a teacher, I did like how students could visually see the map and have an understanding of how different math skills were related and be able to track their progress.  I did not like almost everything else.  First and foremost, students and I could never figure out where to start.  The default for my students, of course, was to start with the easy stuff.  Once things got hard, they would go back up to a different branch they had not been down yet and start with the easy stuff again.  It was extremely difficult to figure out where students should be working or even have any control over what they were working on.  The system was set up here to allow users to simply choose what they wanted to work on and go from there, not to diagnose what was appropriate for them as a learner.

At the start of my third year of teaching, Kahn Academy introduced their current IRT-based model.  After a few years of working my butt off to figure out where my students were and what they should be working on, I finally had a system to do that for me!  It was wonderful.  The first time students logged in, they got their pre-test and were instantly directed to practice skills within that were within a pretty accurate difficulty window for each student.  At first this system was a little unruly since ALL math exercises were clumped into one domain (there’s nothing like consoling a very upset overachieving 14-year-old who suddenly has an integral pop up on their screen), but eventually they split all of their exercises into separate domains based on grade level common core standards and the specific topics we see now.

So now, let’s talk a little bit about what works well and what does not work well about this IRT system in the classroom:

What I loved:

1.  Students ALWAYS had something to do.  The system never stopped recommending what it thought students should do next.

2.  The gamification elements work wonderfully.  Students love collecting badges, points, and unlocking new characters for their avatar (I loved it a little bit too…).

3.  Mastery Challenges are wonderful.  Students have to constantly demonstrate mastery of old skills and if they slip up a few times, the system is quick to re-recommend that practice session.

4.  There are a TON of teacher resources.  Seriously, they have been really thoughtful about how to best support teachers using this system in their classroom.  I was constantly finding new things to reinvest students and help them keep moving.

What drives me nuts:

1.  The tips are terrible.  They are based on simply giving the students a procedure to follow, not based in content at all.  Using a tip also instantly counts a question  as wrong (no difference between using 1 or all of them and have the answer given to you).  Students NEVER wanted to use tips and rarely found them helpful.  Same thing with the videos.  Although this system is good at placing kids, it kind of stinks at teaching them.  Luckily my students had a great teacher to keep them moving.  Although some of my students would login at home and play around, very few of those made much progress without a human coach to work with and go over things they had never seen before.

2.  Although the new domain system is wonderful for directing students what grade level/content they should be working on, the IRT system does not work to place students in different domains.  I had to try to do my own testing and evaluation system to try and place kids in appropriate grade level or skill domains.  Nothing breaks your heart more than telling a 9th grader that they should start off in the “Early Math” section.

3.  If the IRT system decided that students did not know how to do something, it took absolutely forever for my students to prove it otherwise.  Many students would get very frustrated about working through the same problems over and over again that they already knew how to do just because they slipped up once. Similarly, students who got lucky and got a correct answer from guessing would get very frustrated when they encountered topics completely outside of their skill set.  I now know that this comes from the fact that IRT functions under the assumption that a wrong answer means zero mastery and correct answers represent full mastery. Although this assumption seems small, I definitely saw if play out in fairly stressful ways for my students who somehow got misplaced.

4.  Even though there is a lot of data available for teachers, none of it can really be used to show actual student growth and progress on an absolute scale.  As student data is beginning to make its way into teacher evaluations, this is a killer.  Teachers need to be able to show student growth and Kahn Academy just does not currently have that capacity.  It is good at placing students in exercises and getting them working, but does not currently function at the capacity to assess where they are and how much they are growing.

Again, I love Kahn Academy.  It is one of those free education tools out there that I truly believe in.  The addition of IRT into their system has helped in a lot of ways, but definitely has not fixed everything.  While reading and learning about IRT this week, I can now see where those limitations come from, but not necessarily how to work around them.  In the search for the perfect “Teaching Machine,” I think it is important to remember the role of IRT is in placement and not automated instruction.