Thank You #T509Massive!

I just wanted to take a quick moment to thank the teaching team and students in T509 this past semester.  I just submitted my reflection on my online participation this semester and feeling a little nostalgic about how much I learned from documenting my learning on this blog.  This was my first attempt at blogging, and it has been such a wonderful experience.  Looking at my WordPress statistics, my posts have gotten over 200 views since I started writing back in September which is way more than I ever expected.  I definitely felt very intimidated putting myself out there on this blog, but the support I have gotten via comments and in-person conversations have definitely pushed me to keep going.

It has been so great to be a part of a community so willing to take the risk and support each other in making our learning public and visible.  I’m looking forward to keeping up with my blogging as we head into the spring semester and I hope that you all do too so we can all continue to learn from each other!

One Perspective on Urban EdTech Inequality

I’ve written multiple times now about my teaching experience in an urban classroom, but this week’s T509 focus on equity instantly brought me back to thinking about how I saw disparity in Ed Tech implementation play out.  In particular, one quote from Paul Attewell’s article on the digital divide particularly struck me (page 254):

“[Schools] face competing demands from parents, teachers, children, and state testing agencies and suffer severe limitations on spending and staff time.  Schools find it difficult to fulfill their current educational and social mandates, let alone embrace a visionary new one.  The result is often a compromise:  Educational computing adapts to schools at least as much as it “transforms” them (Cuban 1986, 1999).”

In considering this quote, I instant thought of my school district’s Chromebook initiative just last school year.  As part of a grant, every high school classroom in the school district got it’s own cart of 29 Chromebooks.  Instantly, our school became a 1:1 student to computer building after years of functioning with only a handful of computer labs with extremely limited access.  This was one of those moments that the teachers in my building had been wanting for years to improve their classrooms, but the actual implementation of the program left a lot to be desired.

First of all, our building’s current Wi-Fi capacity was not prepared for such a shift.  Even though plans had been put in place to upgrade the network capacity, our district only had an IT team of 3 people who were in charge of upgrading Wi-Fi a all of the district’s 20+ schools while still meeting the high massive volume of responding to IT requests across the district.  As a result, only about 50% of the teachers at my school could access Wi-Fi and only about 50% of them could all simultaneously have a class on the network at a time before the whole thing crashed.  Without Wi-Fi Chromebooks literally become glorified paperweights, and so it was thus for about 50% of the school year.

Second, the launch of our Chromebooks came with a very small amount of professional development (PD).  The IT department (the same overworked and underpaid staff of 3 mentioned earlier) launched a series of voluntary Google apps for education (GAFE) PD sessions with help from a GAFE support person.  This PD pretty much included the GAFE support person teaching us how to login to a Chromebook, access a Google drive, and search for GAFE we were interested in (without providing any real suggestions).  After about only 20 mins of that, the IT staff took over to give us a lecture about how to secure the Chromebooks in our classrooms and how teachers alone are financially responsible for any damaged or stolen Chromebooks in their classroom.

I could go on and on about how Chromebook implementation went wrong in my district, but I think from these two examples you can see what causes some of the gaps Paul Attewell discusses.  One of the key things I want to point out here is that teachers were initially really excited and motivated by the introduction of Chromebooks in our school, but the way in which implementation was handled left most of them feeling dis-empowered and even afraid to use the Chromebooks in productive ways. The only reasons I had any success in this integration was because I was lucky enough to have great WiFi access in my classroom all year and lucked my way into attending some really great outside PD sessions that helped me come up with some creative teaching ideas.

I name this example because I think it magnifies the ideas Justin Reich ends with at around 16:00 in this video.  This is definitely a case in which an increase in Ed Tech innovation is leading to a “Rising Tide” in the achievement gap due to the massive differences in context between low and high income schools.  In thinking about this example, I start to wonder whose responsibility it could have been to “rethink our delivery mechanisms” and prevent poor implementation and create better equity?  Is it on the designers of Chromebooks?  Is it on GAFE developers?  Is it on the overworked district IT guys?  Is it on the school principals and district leadership?  Is it on the teachers who are struggling to find support?  Is it on the people who wrote the Chromebook grant in the first place?  Is it on policymakers who allow for a system that allows for this oversight to exist?  Is it on our society as a whole to start to seek out a major cultural shift?  What if any of this can we actually have an impact on?

Despite my generally skeptical attitude, I really do believe in the transformational quality of open access technologies in education.  As mentioned in this week’s readings and outlined by my Chromebook example, there are a lot of barriers present in our current system that prevent ideal implementation of innovations.  I do not think that intentional technology design alone can break down all of those barriers.  I think we also need to think about how we can make some very simple changes to our education system to allow open access to lead to improved equity.

A Historic Dramatic Rendering of a Dystopian Vision for the Future of Education

Last Tuesday, the wonderful Karen Brennan visited my T600 class and shared with us the image below that I believe perfectly depicts what I fear to be the future of education:

Image from http://www.openculture.com/2012/07/france_in_the_year_2000.html

The terrifying thing is, this picture was created by French artists about 100 years ago as they tried to predict what education in the year 2000 would look like.  It’s fascinating to think about why artists at this time would think of this as a utopian view of education in the future.  I’m more interested in diving into the implications for today.

Three main things concern me about this photo:  The single old white guy selecting content for the machine, the clear uniformity of his students in terms of race and gender, and the massive focus on content delivery instead of experiential learning.  In a lot of ways, this picture depicts what concerns me most about the future of MOOCs, online learning, and other current trends in educational technology.

The same mindset that created the image above is what currently dominates public education.  Major decisions are being made by out-of-touch leaders in education that seem to only benefit a small part of our population.  Those same decisions are also perpetuating the idea that a good education is nothing more than efficient content delivery.  As a result, we have system with massive achievement and opportunity gaps.  While high socioeconomic status students have access to schools and extra resources to supplement their content-riden education with authentic experience, our students in low SES communities are left behind.

I fear that current trends in education innovations are only perpetuating this gap and not considering how public education as a whole needs to shift.  I believe we need to start using innovation to challenge the ideas that this 100-year-old picture represents.  Our students deserve a system that gives them a voice in what content is important for them, allows for adaptation to diverse audiences, and emphasizes experiences and performances of understanding over content digestion and regurgitation.  We need to not just think about how to bring education to a large scale, but how to bring the right type of education to scale.

Reflections on Unbundled K-12 Education

After hearing from Anant Agarwal discuss his article on “unbundling” higher education during T509 last week, I have found myself spending a lot of time thinking about the implications of an unbundled K-12 education system.  Based on our class discussion, the Education Reform for the Digital Era report, and Justin Reich’s blog post, I have been feeling a little torn about this potential school system of the future.  The ideal implementation would truly open kids up to many new education opportunities and allow for personalized learning, but when has any implementation in education been ideal?  What types of benefits could we expect from an unbundled school system and what should we be skeptical about?

For maybe the first time on this blog, I am going to start by talking about the potential positives I see in an unbundled school system.  When first discussing the idea of unbundled schools, I had a really hard time thinking about what the actual benefits are in this system.  Today, however, I had a wonderful opportunity to visit NuVu Studio in Cambridge.  Started by architect Saeed Arida, NuVu Studio is a program that takes students out of their traditional school settings for 12 week periods during the school year to experience education in a new way.  Students at NuVu are organized into design teams to undergo rigorous problem-based learning focused on solving real-world problems in their community through an iterative design process.  In their 12 weeks, students take on several projects in 2-week chunks to learn how to navigate the design process in an attempt to solve large open-ended problems. The results are absolutely amazing and you should all definitely check out their website to learn more.

What was extremely interesting to me was the logistics behind NuVu.  When NuVu accepts a student, they partner with that student’s school to determine what needs to be done for that student to take-on this 12 week removal from traditional school.  Many of their students simply need to take a 1-2 online courses to augment their learning during those 12 weeks simply to meet legal requirements, but Saeed explained to me that teachers and school administrators have frequently been blown away by the amount of progress their students make while taking on projects at NuVu.

To me, this whole idea is the closest thing I have seen to unbundled education.  NuVu is a private, for profit company that has developed an extremely impressive education model that adds enormously to students’ educational experiences.  When thinking about why an unbundled school system could be desirable, this type of model definitely makes me second guess a lot of doubt.  It would be amazing to see the NuVu model scaled and made accessible to all students using their “backpacks” of education funding.  After my visit, I am now finding myself daydreaming about what other models could appear in an unbundled system.

Always the skeptic, however, I can definitely see the potential for unbundled school systems to do more harm than good.  Specifically, I believe the proposed models I have read through would definitely serve only to widen the opportunity gap we currently see between students in low and high-income communities.  The model presented in Education Reform for the Digital Age proposes that such gaps could be alleviated by giving students in low-income situations access to more funds in their  “funding backpack.”  They are making a key assumption here that I believe is fundamentally flawed:  the only thing preventing low and high-income students from performing at the same level is money.

From my experiences, money is only a very small part of the very large problem concerning education inequity.  My students who struggled with school encountered many problems including home environments not conducive for studying and doing homework, major family obligations such as babysitting or working part time to help pay bills, various traumatic experiences leading to emotional instability, and a lack of positive role models in their community to message the importance of education.  Simply unbundling a school system and allowing more choice in educational experiences will not take away the vast majority of these outside issues hindering quality education.  In the current system, schools already do not get enough funding to provide services for students with such profound needs.  I find it hard to believe that a new system would be able to increase funding per student substantially enough to make that difference.

I can also picture a world in which low-income communities would rapidly become educational deserts.  In providing funds for students to seek education elsewhere, many low-income students may take that opportunity to travel away from the historically failing schools in their communities to higher performing schools in neighboring locations.  When those schools inevitably reach capacity, what options will still exist for students?  As local failing schools inevitably fail due to lack of funding, how will students navigate new problems caused by a lack school options in their direct neighborhoods?  What new educational opportunities will form and how many of them will take the risk to enter low-income communities based on modern mindsets?  At what point will online learning or low quality education programs become the only option for some students in a system that was supposed to open-up opportunities?

As stated before, I definitely see how a world of unbundled K-12 education could have massive positive impacts on our current system.  I also wonder how such an innovative system would actually allow for removing substantial barriers to equity and access in today’s society.  How do we ensure that an unbundled education creates high quality education experiences for all learners and does not simply widen the gaps that already permeate the education landscape?

What’s Worth Scaling?

I kicked off this week’s T509 prep with Anant Agarwal’s TED Talk on MOOCS and blended learning.  Towards the end of the clip, he calls the audience to take a moment to dream about education in the future.  He mentions that he can imagine a world where a college campus has only one lecture hall simply standing as a reminder to our grandchildren that we used to learn by sitting in a room and listening to a single lecturer without the convenience of a rewind button.  This all comes after a discussion on how content created through MOOC production is revolutionizing the way we can think about blended learning.  He points out key features such as videos capturing great teachers explaining complex topics and interactive assessments providing instant feedback and personalized learning.  While listening to him discuss a future of blended learning, I had to stop for a second and think about my own vision for my grandchildren’s education.  What would I want education of the future to look like and how do MOOCs influence that future?

In my T600 Class: Thinking and Learning Today and Tomorrow, we have been working on applying research ideas from Project Zero to thinking about what good education looks like.  I do not think that at any time one of my classmates has brought up that what educators need are really good lectures or self-graded assessments.  We have spent a lot of time, however, discussing how to create student-centered performances of understanding that guide learning and make it visible to students and teachers.

When I was teaching, I established a practice of having my students write letters to my future students as soon as they finished their final exam.  The assignment was for them to give that student advice and let them know what to expect in my class.  I always loved diving into reading these letters so that I could reflect on my practice from that year and get a glimpse of what my students thought about my class.  Not once did a student ever write anything like “His lectures are amazing” or “His quizzes really make me think.”  Instead I would read things like “He helped me understand that math is everywhere” or “I now know that math is used all the time in the real world.”  These types of responses did not emerge because I was really good at talking and writing good tests, but because I was focused on creating projects for my students that helped them learn math through application and reflection.

In thinking about the content of MOOCs (specifically xMOOCs), I can only think of two key elements: short video lectures and some form of automatically graded assessment for instant feedback.  All of these features tools for scaling pretty standard education practices in education today:  lectures and assessments.  By scaling these practices, are we really transforming education?  In my experiences, absolutely not.  I believe that teaching and learning worth scaling are practices that encourage students to experience authentic projects and engage with material in individual ways.  MOOCs in their current form are simply translating old practices to a new medium, not changing the same education practices that have led to major calls for reform.

In this week’s readings, I was inspired by a few key efforts to scale education that align with my vision of experiential learning.  I was impressed with the New York Times article on how Harvard Business School is approaching online learning.  Instead of jumping on the bandwagon of posting video lectures, Business School leaders went about thinking how they could recreate their case study based teaching and class cold calling into online learning.  On page 15 of the Institute –wide Task Force on the Future of MIT Education: Final Report, recommendations are given to MITx to think about ways to bring problem-based learning strategies to scale.  One incredible idea given as an example involved creating a series of MITx courses around air pollution to bring MOOC learners together around developing solutions in their communities.  These two examples captured for me what I think excellent learning at scale should look like.

When it comes to transforming education, I think many MOOCs are currently only thinking about a small part of the problem.  They are providing education and creating content at a large scale, but is it the type of education and content we should be scaling?  In future blended learning classrooms, it would break my heart to see students simply sitting around watching videos and taking assessments to show mastery.  I would want to see my grandchildren in an education system where they are engaged in meaningful projects and showing mastery through performance, not regurgitation.  When it comes to making that system a reality, I think MOOC advocates should be thinking less about how to better scale presenting and assessing content.  Instead, we should be thinking about how to scale practices that foster authenticity and engagement in learning.

Why Every Teacher Should Make a MOOC

It took me a little longer than usual to think of what I wanted to write about this week for T509.  This week’s readings took us on a deep dive into the MOOC production process and we had several HarvardX and edX folks come and speak to our class about what it’s like to work with faculty to produce one of these online learning experiences.  Going into the class, I had one key question.  According to The Chronicle of Higher Education article, MOOCs have shown very little success in terms of outcomes for students and MOOC professors seem fairly split on whether or not their courses will ever actually be recognized for credit at major institutions.  Yet, the vast majority of MOOC professors reported that they believe MOOCs are “worth the hype.”  I couldn’t help by wonder:  If MOOCs do not seem to hold much promise in progressing student outcomes and lowering barriers to higher education, why in the world do MOOC professors think they are so awesome?

Enter Annie Valva, Associate Director of Instructional Development at HarvardX.  In a matter of minutes of talking to our class, Annie had me at the edge of my seat as she answered a very tough question about HarvardX MOOCs:  Why is Harvard even doing this?  She summed up her response with her “3 R’s:”

Reach – To increase the number of people who have access to courses by prestigious Harvard faculty beyond campus and the USA.

Research – Each MOOC course generates potentially millions of data points about online learning that can help uncover answers to big questions about new education trends.

Recirculation – Professors who make MOOCs take their key learnings and new media resources back to their normal “residential” classes to improve teaching in learning on campus.

As you can tell from the bold, the third “R” was the one that really caught my attention.  Annie herself emphasized the importance of this normally unseen goal.  Annie went on to discuss many anecdotes of notable professors who stated many ways in which MOOCs impacted the way that they think about teaching and learning in all settings, not just online.  Clearly, this got me thinking about why in the world would making a MOOC have that kind of impact on how college professors think about teaching.  I have a few theories:

1.  The MOOC design process requires professors to think deeply about course design.  Annie discussed with us how professors at HarvardX have to go through a rigorous process to outline their course and think about pedagogy before HarvardX will start making their course a reality.  She found that when first asking professors what their learning goals were for the course and how they would measure learner outcomes, many of them responded with a blank stare.  This was the first time for many of these professors in the decades they had been teaching that they were asked to stop and be purposeful about what they were teaching and what they wanted students to learn.  Not only that, but Annie also mentioned that professors were constantly referred back to those goals and outcomes when making small decisions in the process (such as what questions to ask or what to include in a short video).

2.  MOOC professors had the opportunity to work with an diverse interdisciplinary team with innovative resources to make their course a reality.  HarvardX’s team includes software engineers, media producers, film editors, a talent coaches, and many other skill sets to make a professor’s vision come to life.  With this access, a professor actually has a powerful team and set of resources to use to find innovative ways to present materials and measure student outcomes.  They are also supported through the whole process by individuals from various backgrounds to constantly present new ideas and give feedback on course choices.

3.  MOOC professors are designing for a way of learning that they know very little about.  Every teacher at one point has been a student in a classroom, but a much smaller number have ever been a student in an online course with multiple thousands of classmates.  When designing their courses, MOOC professors are thinking about reaching and retaining an audience of learners that they are really unfamiliar with.  It’s amazing to me how such a small shift suddenly leads to choices such as “videos can only be about 10 minutes long” and “we need to include constant formative assessment to see how they’re doing.”  In reality, those teaching choices are not much different than good practices that are frequently ignored in traditional classrooms.  It’s interesting that, by simply making the audience seem unfamiliar, it becomes easier to adopt teaching practices that seem so heavily resisted in familiar environments.

These are only a few of the theories flying around my head, but this brings me to the title of this post:  every teacher should make a MOOC.  Now, clearly I’m not really saying that all teachers in the world should go out and haphazardly attempt to make a class to educate thousands of learners.  There is something going on here, however, that I do not think educators should ignore.

When thinking about issues in our current education system, both higher education and K-12, teacher quality is something that comes up time and time again.  As a young teacher myself, I constantly struggle with finding ways to improve my teaching and turn my classroom into a place where deep learning is happening.  It sounds to me that making a MOOC would be an amazing way for me to develop my teaching skills.  I would love the opportunity to work with a diverse team to develop a course I care about from start to finish and deeply think about how my choices affect student outcomes.  I would love to team up with media specialists to create videos and animations that bring learning to life in the classroom and create learning experiences through technology.  Most of all, I would love to launch my course ideas and see how they work with a “massive” audience.  In a way, that massive size of the audience and the nature of MOOCs would make the whole experience fairly low-stakes.  If my course stinks, none of my learners would not really miss out on anything since MOOC completion currently counts for very little and now I would have a platform to rethink teaching for a more high-stakes audience.  In the process, I would have learned a ton about what innovative teaching practices look like so I can apply them to my classroom practice.

In reality, these elements of deep thinking about course design, interdisciplinary collaboration, and an unfamiliar low-stakes audience seems like an ideal recipe for an amazing teacher preparation course.  These ideas have me thinking not about how MOOCs may change the educational landscape for students, but how they can change it for teachers.

I’ll admit, the idea of scaling the creation of massive education is slightly absurd, but I do think there is something we can learn from MOOC production about how we can start improving teacher quality.  Aside from my small theories developed during my morning T ride, it would be interesting to really take a look at what happens during MOOC production that results in big changes in teacher mindsets about teaching and learning.  We should take advantage of the current hype around MOOCs to answer a new question:  how do we scale a teacher’s experience of scaling education?

Thoughtless vs. Thoughtful Blended Learning

This week’s T509 reading theme on Blended Learning really got me thinking about how different strategies mentioned play out in real schools with real kids.  I enjoyed exploring the case studies featured during the Khan Academy Blended Learning module and seeing good practices play out in some fairly innovative schools.  However, I couldn’t help but notice that the featured schools were all charter schools.  This means that a degree of their success with blended learning definitely needs to be partially attributed to the fact that public charter schools benefit from their ability to function mostly outside the current system of education in the US.  Their school leaders have much more freedom than the typical public school when it comes to making decisions about how to allocate their funding and design their school day and calendar.  As seen from the case study videos, these freedoms allow them to make thoughtful decisions about how to effectively implement blended learning practices to enhance education for students.

So what does this mean for public schools trying to implement blended learning systems?  Let me give you a case study from my own teaching experience.

As mentioned in the Clayton Christensen Institute White Paper, one current use for blended learning is for credit recovery for high school students.  The school I was teaching at was using a popular credit recovery system for a night school program.  The purpose of night school was to allow students who could not attend regular day school for various reasons an opportunity to earn their class credits and graduate through this alternative program.  Essentially, the students were given a list of courses they needed to complete and could work through an online module for each of those courses with the assistance of teachers hired to proctor during night school hours.  Even though this model caught on quickly, there were a lot of problems with it.

First of all, night school became a dumping ground and a replacement for true interventions.  If a kid keeps getting in trouble in normal classes, send him/her to night school.  If a kid is failing all of his/her classes, send him/her to night school.  If a kid is over age and under-creditted, send him/her to night school.  The goal of night school moved away from being accommodating to students with exceptional life circumstances and became a place to send all of the “undesirable” students rather than actually attempt any personalized interventions.

Second, night school instruction was questionable.  To be clear, I never taught a night school class and only heard from other teachers what the actual learning climate was like.  That being said, I heard over and over again about students who never watched or read through any of the instruction material.  They simply clicked through screens until they got to assessments and googled to find answers.  Even in rooms where teachers did not allow that practice, instruction from the computer relied on basic “read this” followed by “now answer these questions” approach no different than many textbook-style education methods.  Students never had the chance to engage in any activities, projects, or even class discussions to augment their learning.  It was all basic regurgitation aided by a teacher who frequently did not even share a background with the content that a student was “learning.”

Third, many students in the night school program did not have the basic skills they needed to access the material.  As mentioned before, the selection process for students was focused on getting “bad kids” out of the normal school day rather than the right kids into a blended learning environment.  Many of the “bad kids” are the ones with the biggest learning challenges and are the most behind.  One of my former students, let’s call him John, fit this situation.  John was a giant pain in my butt in my 9th grade Algebra class and was constantly getting in trouble everywhere he went.  It did not take long for all of his teachers to realize that 15-year-old John was completely illiterate.  He couldn’t read or write anything at all.  His attitude and behavior problems were clearly steaming from his inability to function academically.  We communicated our finding to all of the appropriate levels, but then John turned 16 and he was eligible for night school.  Rather then take any steps to help join and intervene, the school simply sent him on to this blended learning environment that mostly required him to read to learn content independently.  He stopped attending night school in less than a month and dropped out of high school by the start of the next school year.

Finally, even the kids who excelled in night school and earned their diploma encountered a rude wake-up call upon graduation.  The vast majority of public universities would not accept the credits they earned from the online learning platform.  Instead of broadening horizons for these students, the night school program in fact significantly limited students’ choices when it came to furthering their education.  Their only options now were to attend community college or try to find a private school that would accept their credits but was financially beyond their means.  The worst part of this was that NOBODY told the students in night school that this would happen until it was too late.

My point here is not that blended learning is awful and leads to kids dropping out and not getting into college, but that implementation models matter.  When talking about blended learning models, it’s easy to tell the success stories when things work well.  What we tend to forget is to tell the stories of when things go wrong and what that means for kids and their potential futures.  I can only speak for my experience in my public school system, but I am sure there are similar stories out there of thoughtless implementation of blended learning to try and solve a problem.

If we are to accept that blended learning models are “disruptive” and will ultimately take over traditional education models, then we need to start thinking about how we can build capacity in public school systems to thoughtfully implement these models to benefit kids and improve outcomes.  Part of that thoughtfulness needs to be focused what blended learning needs to look like in public schools to account for the many limitations in our current system.  Rather than working quickly to try and keep up with the hype, schools and teachers need to take their time and be thoughtful about their adaptation of blended learning.  In some cases, this solution may not even be an appropriate match for their problems.

Big Data, Automated Essay Scoring, and Student Diversity

I was extremely excited to draw some connections between two of my classes this week, T510S – Data Science in Education and T509 – Massive: The Future of Learning at Scale.  The key to this link: when technology gets big, how do you think about ensuring fair representation of diverse populations?  I will be first to admit that this connection seems a bit obscure, but stay with me here and I promise I’ll get you there.

First, lets talk about T510S.  We had a wonderful HGSE Ed.L.D guest speaker (digitally) join us to discuss ways in which big data has the potential to completely exclude or misrepresent diverse groups.  She provided some awesome examples including the City of Boston’s Street Bump app.  This innovative app runs on smartphones and collects “bump” data while the user drives to determine what streets need repair.  What a great way to collect data, right?!  Well, as discussed at the end of this article, the app quickly led to many street repair reports coming from wealthier neighborhoods in Boston while less privileged neighborhoods received less attention.  A general assumption was made in the implementation of this program:  Everyone in Boston had access to a smart device needed to make this data robust.  In reality, gaps in access literally led to gaps in service.  This problem has since been addressed, but I think it is an important case in the type of inclusive mindset that is too often ignored in innovation.

Our T510S guest shared other cases in Big Data discrimination and noted some overall trends.  As a society highly focused on collecting data and information, it’s important to keep in mind that different groups of people contribute to data in different ways and amounts.  We need to remember that the narrative that data tells has not been equally contributed to.  She left us with the advice to actively think about the stories that are missing from a data set as well as the ones that are present.

Given that T510S meets on Monday night and I have been habitually tackling my T509 readings on Tuesday afternoons, these ideas were still very present in my brain when reading about Automated Essay Scoring (AES).  While reading Justin Reich’s Ed Tech Researcher post and Shermin’s research supporting the validity of AES scores, I asked myself “What story is this data missing?”  I found my answer on page 27 of Shermin’s paper:  “An important aspect of system performance to evaluate before operational use is fairness – whether subgroups of interest are treated deferentially by the scoring
methodology.”

When looking at Shermin’s data collection (Table 1, page 31), I noticed that the demographics of captured by his data sets represented populations of either majorty white or close to 50% white students and at most 46% free or reduced lunch students.  Why does this matter?  AES programs are based on comparing unscored essays to a set of already scored essays and assigning scores based on features that are similar.  So what happens when the scored essays being used as benchmarks for the system represent students whose backgrounds do not necessarily match the backgrounds of students being graded?

As an urban educator, I worked at a school where 50% of students are hispanic, 44% of students are black, and 99% of students receive free and reduced lunch.  When thinking about my kids and the essays they could potentially write to be graded by an AES system, I worry about the potential discrepancy between their writing and the benchmarks a machine is comparing them to.  I started thinking about the role that factors like cultural differences in language use and English language learning could have on a machine’s perception of their writing.  How can we make sure that AES systems can account for those differences and not leave them out of the picture of what a “good essay” looks like.

I did a quick google search and came across this annoted bibliography from the Council of Writing Program Administrators on AES systems.  Section 4 of this document covers the role of diversity in AES systems.  I was relieved to see that my question about diversity is being questioned by many.  From a quick skim of the different studies, it seems as though results are fairly mixed but there is evidence suggesting differences in AES scoring across student subgroups.  I think it is so important that this research is continued and expanded on to ensure that AES systems can address these disparities before wide implementation.

In summary, I have learned this week that it is very easy for innovation to get lost in the “bigness” of all of the newest data and technologies and forget about the smaller impacts generalized solutions have on specific people.  In education in particular, we are working in a system with a strong history of implementing “solutions” that leave out large populations of students because they don’t behave like the “average.”  I’m definitely interested in looking more into what we can uncover if we continue to look at data and ideas  through a lens of “who is missing?”

My Experiences with Khan Academy and IRT in the Classroom

First of all, I absolutely loved the readings, videos, and assignments for this week’s T509 class.  All of the content was all around really awesome stuff and full of fascinating ideas.

I could not pass up the opportunity to write about some of my experiences with Kahn Academy.  I am a huge fan of their system and have used it in my math classroom for the past three years.  I also tend to find myself spending an hour or two here or there practicing my own math skills as well (kind of really proud of my Profile).

To establish some context, I started out my career teaching a 9th grade math intervention course.  I was plopped in a computer lab with absolutely zero curriculum tools and given the most behind 9th graders and told to do something about it.  After some testing, I found that my students were on average at a 5th grade math level and needed a lot of support if they were to be successful in Algebra 1.  Kahn Academy was one of the first places I went to to find solutions to this problem.  Back then, the interface looked something like this:

From http://www.freelyeducate.com/2012/03/free-math-program-from-khan-academy-.html

Students could navigate the “knowledge map” at their own pace.  Blue squares represented mastered skills, green recommended, and orange was a previously mastered skill that the system believes needs to be reviewed.  From my understanding, this system was not grounded in IRT at all.  It simply displayed a curriculum map under the assumption that completing one skill meant that the student was ready to move on to the next and if they struggled with that next skill, then they should go back and review the skill that came before it.

As a teacher, I did like how students could visually see the map and have an understanding of how different math skills were related and be able to track their progress.  I did not like almost everything else.  First and foremost, students and I could never figure out where to start.  The default for my students, of course, was to start with the easy stuff.  Once things got hard, they would go back up to a different branch they had not been down yet and start with the easy stuff again.  It was extremely difficult to figure out where students should be working or even have any control over what they were working on.  The system was set up here to allow users to simply choose what they wanted to work on and go from there, not to diagnose what was appropriate for them as a learner.

At the start of my third year of teaching, Kahn Academy introduced their current IRT-based model.  After a few years of working my butt off to figure out where my students were and what they should be working on, I finally had a system to do that for me!  It was wonderful.  The first time students logged in, they got their pre-test and were instantly directed to practice skills within that were within a pretty accurate difficulty window for each student.  At first this system was a little unruly since ALL math exercises were clumped into one domain (there’s nothing like consoling a very upset overachieving 14-year-old who suddenly has an integral pop up on their screen), but eventually they split all of their exercises into separate domains based on grade level common core standards and the specific topics we see now.

So now, let’s talk a little bit about what works well and what does not work well about this IRT system in the classroom:

What I loved:

1.  Students ALWAYS had something to do.  The system never stopped recommending what it thought students should do next.

2.  The gamification elements work wonderfully.  Students love collecting badges, points, and unlocking new characters for their avatar (I loved it a little bit too…).

3.  Mastery Challenges are wonderful.  Students have to constantly demonstrate mastery of old skills and if they slip up a few times, the system is quick to re-recommend that practice session.

4.  There are a TON of teacher resources.  Seriously, they have been really thoughtful about how to best support teachers using this system in their classroom.  I was constantly finding new things to reinvest students and help them keep moving.

What drives me nuts:

1.  The tips are terrible.  They are based on simply giving the students a procedure to follow, not based in content at all.  Using a tip also instantly counts a question  as wrong (no difference between using 1 or all of them and have the answer given to you).  Students NEVER wanted to use tips and rarely found them helpful.  Same thing with the videos.  Although this system is good at placing kids, it kind of stinks at teaching them.  Luckily my students had a great teacher to keep them moving.  Although some of my students would login at home and play around, very few of those made much progress without a human coach to work with and go over things they had never seen before.

2.  Although the new domain system is wonderful for directing students what grade level/content they should be working on, the IRT system does not work to place students in different domains.  I had to try to do my own testing and evaluation system to try and place kids in appropriate grade level or skill domains.  Nothing breaks your heart more than telling a 9th grader that they should start off in the “Early Math” section.

3.  If the IRT system decided that students did not know how to do something, it took absolutely forever for my students to prove it otherwise.  Many students would get very frustrated about working through the same problems over and over again that they already knew how to do just because they slipped up once. Similarly, students who got lucky and got a correct answer from guessing would get very frustrated when they encountered topics completely outside of their skill set.  I now know that this comes from the fact that IRT functions under the assumption that a wrong answer means zero mastery and correct answers represent full mastery. Although this assumption seems small, I definitely saw if play out in fairly stressful ways for my students who somehow got misplaced.

4.  Even though there is a lot of data available for teachers, none of it can really be used to show actual student growth and progress on an absolute scale.  As student data is beginning to make its way into teacher evaluations, this is a killer.  Teachers need to be able to show student growth and Kahn Academy just does not currently have that capacity.  It is good at placing students in exercises and getting them working, but does not currently function at the capacity to assess where they are and how much they are growing.

Again, I love Kahn Academy.  It is one of those free education tools out there that I truly believe in.  The addition of IRT into their system has helped in a lot of ways, but definitely has not fixed everything.  While reading and learning about IRT this week, I can now see where those limitations come from, but not necessarily how to work around them.  In the search for the perfect “Teaching Machine,” I think it is important to remember the role of IRT is in placement and not automated instruction.

Connectivism: I don’t get it.

While reading a few articles on connectivism for tomorrow’s class, I have to admit that I just do not get it.

Essentially the idea is that learning has nothing to do with actually knowing anything, but building connections and capacity for learning in the future. The arguments (at their most extreme) go as far as to claim that education should be restructured so that students study only what they are interested in and go about that study through working with peers and taking advantage of open online resources. In a sense, teachers are no longer teaching content. Instead, they are teaching students how to access existing knowledge to facilitate their own learning and add their own voice to that body of knowledge.

Now, call me old-fashioned, but I think these ideas are a touch absurd. Yes, in theory everything sounds wonderful and jolly and so much fun. I am having a very hard time, however, imaging what such a system would actually look like. In the end, what are students actually learning? What does success in a connectivist classroom actually look like? What outcomes do we look for that show student learning is real? How does connectivisim provide students with real future opportunities in actual society?

Not only do these ideas seem to me to be unrealistic, but also biased. I find the insinuation here that everyone in society has the capacity to uncover their true interests and deeply study those interests by forming connections with similar minded people in their community and on the web. Yes, these ideas sound like they would work wonderfully in a community full of strong role models and resources for learning. We all know that such communities are far from universal. Not only that, but many students in today’s society lack many of the most basic skills needed to be able to access quality online learning resources and direct themselves in their own learning. The connectivist theories I have read about this afternoon seem to apply only to the most privileged in our society and leave the others behind.

In my opinion, content is crucial in learning. If our mission is for students to be able to grow into contributing members of society, there are large sets of knowledge and skills they need to possess. I agree that good teaching includes growing students’ capacity for learning, but I do not think that importance overshadows what content students actually understand and can apply.

Fellow T509ers, what are your thoughts? Am I missing something? Has anyone found resources outside of the readings (I even tackled some of the rabbit holes!) that may clarify my confusion? What do you think successful connectivist teaching really looks like?