The Trouble with Value Added Models: Part One

Although I'm an educational researcher, I'm not given to weighing in publicly on various policy debates. It's not that I don't have opinions; it's just that all the yelling and ad hominem fallacies are really, really boring.

That being said, I am a professor, and every now and then I should probably profess something. So - for the record, I am adamantly opposed to Value Added Models of measuring teacher effectiveness.

For those who don't know, Value Added Modeling (or VAM as I'll call it from here on out) is essentially a way of looking at student test scores and trying to figure out how much of those scores are due to the teacher. In theory, students with better teachers will perform better than they did the year before, and they'll perform better in relation to other students with other teachers. VAM seeks to use all that comparative data to isolate each teacher's unique contribution to students' scores. This way, we can measure how much value a teacher has added to students' educations, and we can make staffing and/or training decisions accordingly.

On the surface, this is not a bad idea. Teachers are responsible for student learning, and we are absolutely justified in creating and establishing systems that assess teachers' effectiveness. Student performance data is a vital and valuable part of this process. In fact, teachers already use student data to evaluate and adapt their own teaching. In Washington State, the edTPA, TPEP, and ProTeach (not to mention National Board Certification) all operate according to this central premise: effective and proficient teachers will establish and use systems for collecting and analyzing student data, and they will use their analyses to adjust and refine their own practices. TPEP, which is the official and mandated evaluation system used by school districts, adds a supervisory component to this. It's not enough for a teacher to be able to do this on her own; she has to be able to prove it to her boss.

So it's not like teachers aren't using student data, and it's not like they're not being held accountable for student performance. In that sense, it's possible that adding VAM to teacher evaluations might be overkill, but that's not really the problem.

The trouble with VAM is twofold: 1) it's being used in really stupid ways; 2) it's not necessarily measuring what it claims to be measuring. It's the latter point that I want to address right now.

Increasingly, research is demonstrating that VAM derived measures of teacher effectiveness might not be assessing the teacher as much as they're assessing the demographics of the students in the classroom. For example, Newton, et al (2010) analyzed various Value Added Models of teacher effectiveness and determined that regardless of the model used, and regardless of which variables they controlled for statistically, the demographics of the students always had a significant correlation with the teacher's effectiveness scores. Specifically, as the number of ELL students went up, the teacher's effectiveness scores went down. Again, this was even after they had statistically controlled for ELL, and it happened no matter which Value Added Model they were using. Their conclusion:

"We simply cannot measure precisely how much individual teachers contribute to student learning, given the other factors involved in the learning process, the current limitations of tests and methods, and the current state of our educational system."

I could go on, but that sums it up pretty well. Granted, the arguments both for and against VAM are more complicated than I've outlined here. There are some compelling arguments in favor of it (e.g. it works if you measure student year-to-year gain instead of using static measures), and there are some compelling counterarguments to those arguments (e.g. even something like rates of learning might be co-variates with certain demographic measures). In any case, we have both theoretical and empirical reasons to doubt the suitability of VAM as a measure of teacher effectiveness. The problem isn't that we're trying to hold teachers accountable: the problem is that we may be trying to hold them accountable for something that is simply out of their control.

So, of course, it makes perfect sense that the federal government would mandate VAM as part of each state's teacher evaluation system, and it makes even more sense that they would try to extend VAM to use student test scores as a measure of the value added by the teacher's preparation program.

That's right: the Department of Education is pushing to use K-12 student test scores to measure and rank the effectiveness of teacher education programs. We'll talk about this particular stupidity in part two.

I have a new article out in the Australian Journal of Teacher Education. It's the first in a series of articles I have coming out from my most recent line of research.

Here's the short version:

- Person A wants to be a teacher. Person A also has a pretty clear set of expectations about what that's going to be like.

- Those expectations are most likely wrong. We profs call them "misaligned."

- If Person A doesn't revise her expectations, she's going to run smack into a brick wall of reality once she enters a classroom. It's going to hurt. It's going to be disillusioning. And, unfortunately, it's probably going to result in her leaving the profession early. This smackdown with reality is referred to as "practice shock."

- So, teacher preparation programs like the one I work in will do their students a great deal of good if they can help them confront and revise their expectation before they get in the field. That way, when they do experience practice shock (because there's no way to avoid it completely), they'll be able to use it as a platform for growth instead of become disillusioned and burning out.

But how can preparation programs do that? That's what my research is about.

Here's the article. Remember, this is just the first step in a long research process. I don't have all the answers. Heck - at this point, I don't even have all the questions.

And here's a picture of a kangaroo, because, you know, Australia.

Thank you to the faculty and staff in the School of Education at Seattle Pacific University. I couldn't be prouder to call myself an SPU alumnus, and I'm humbled and honored.

Teaching: Three Simple Words

The inner sanctum

Teach.

Adjust.

Repeat.

How hard can it be?

I'm an assistant professor of education. I have a doctorate in curriculum and instruction from a well-respected institution. I have years of teaching experience at the both the high school and collegiate levels. I've published in the right journals and presented at the right conferences. I'm the go-to guy, at the head of the class, and, if you squint your eyes real tight, I'm an expert in my field.

But here's the truth: I'm still a beginner.

This is what years of professional experience have taught me: teaching is hard. Though aspects of it get easier, it never gets easy, and mastery is always out of reach. It's a profoundly frustrating endeavor, particularly in a consumer based culture, because the pay off moment when the perfect student rolls out the factory door never arrives. It's like running a marathon where the finish line moves at the same pace you do. You sweat and agonize for hour upon hour, only to discover that you haven't left the starting line. There's no summit to reach and no final clue to decipher. There's no pot of gold at the end of the rainbow, because the rainbow never ends. It's a road trip with no destination, and if you ask "are we there yet?", the answer will always be "no."

And that's exactly like it should be. This is both the simple truth and profound mystery of teaching. Expertise isn't a matter of getting it right; it's a matter of getting it wrong in a systematic fashion. It's not about knowing all the answers. It's about being hyper aware of what you don't know. It's about making plans based on the best information that you have and then gathering new information in a rigorous and reliable fashion. It's about absolute fidelity to your one little corner of the puzzle even though you'll never get to see what it looks like when it's finished.

So teaching is hard. I tried to find a witty wrap-up for this post, but that pretty much sums it up right there.

Fathers and Sons: Star Blazers - PART 1

On April 7, 1945, the flagship of the Japanese navy was sunk by US torpedoes off the coast of Okinawa. She was the heaviest and most well-fortified naval vessel the world had ever known. When she sank to the bottom of the Pacific ocean, she carried with her not only the lives of 3000 of her crewmen but also the hopes and dreams of an empire. She was the Yamato, and her fate and mine have been entertwined for the last 30 years.

I don't remember when I fell in love with stories, but I do remember when I discovered Star Blazers. I was six years old, and every day after kindergarten I would fly down the stairs to the basement, turn on the TV, fiddle impatiently with the rabbit ears, and settle back to let the sweet sights and sounds of black and white, monophonic science fiction wash over me. Though many of the plot's subtleties were lost on me at the time, I had a sophisticated enough grasp of the narrative to understand the basics:

It's 2199, and the Earth is under attack. The Gamilons, a race of humanoid aliens from distant space, have destroyed their own planet and set their sights on Earth as a replacement. From their base on Pluto, the Gamilons have deluged Earth with radiation bombs. The entire surface of the planet is uninhabitable, and the remaining Earthlings have evacuated to underground cities. In a little less than a year, the radiation will penetrate the surface, and all human life will be destroyed.

Earth's military forces have been obliterated. In fact, only one deep space battleship remains operational, and even it has been heavily damaged. Just when it seems that all hope is lost, Earth's commanders receive a mysterious message from a woman named Starsha on the distant planet Iscandar. She has a machine - the Cosmo DNA - that can remove the radiation from the planet. But Iscandar is light years away, and time is running out.

Only one hope remains. An old naval vessel, a battleship sunk in a long ago war, has been retrofitted as a starship. Starsha has sent the Star Force the technical plans for a new kind of engine that will allow them to travel faster than the speed of light. Will this new "wave motion" engine allow them to travel to Iscandar and back in time? Will they be able to defeat the Gamilons and save the Earth?

This is my story, the first epic I remember learning. It's possible that I had internalized some biblical epics by this point in my life, but they never had the visceral hold on my imagination that Star Blazers did. My sense of good and bad, of heroes and villains, of loss and honor, were all shaped by the fertile imagination of Japanese animator Leiji Matsumoto.

Though Star Blazers first aired in America (dubbed into English) in 1979, it was already well known and loved in Japan as Yamato. In 1974, audiences in Japan were thrilled and captivated by the story of the young heroes who resurrected the great battleship in order to defeat the enemy and his radioactive bombs. Though American audiences saw it as a well crated space yarn, Japanese audiences saw it as a love song for a fallen empire: the great Yamato - which, by the way, is a historic name for Japan - rises from the ashes of its destruction and soars off to glory once again. Less than 30 years after the twin bombings of Hiroshima and Nagasaki, the main cultural export from Japan to the US was the story of the Yamato, coming back from the dead and using superior technology to defeat the foe that had crippled them not long ago.

But all this cultural subtext was lost on me as a child. All I knew is that they were in space, they needed to save the world, and they only had 347 days left in which to do it. What did I care about resurgent nationalism and the ghosts of empire? Nothing.

Nothing, that is, until I introduced it to my kids last September.

More in part two.

Fathers and Sons: Star Blazers - PART 2

The trouble with me is that I can't just let a story be a good story. I have to pick it apart, analyze it, look under the hood, kick the tires, and then dismantle the manifold just so I can put it back together again. It's a frustrating habit, one born out of years of teaching students to do the same, and as much as I value this archeological approach to literary understanding, I sometimes wish I could just turn it off.

My family feels the same way.

We downloaded the first season of Star Blazers from iTunes last September. "This is so cool," I told my wife while I was forcing her to watch the first episode. "It's a show that both repudiates and celebrates Japan's imperialist heritage. By rebuilding the Yamato they are reforging their collective identity, one based on global, not national, interests."

She yawned.

"No, think about it," I continued, bouncing up and down a little bit in my excitement. "This came out 30 years after WWII ended. It's a new generation of Japanese, their equivalent of the baby boom. They've inherited their parents' pain and disappointment, and yet it's been tempered with the optimism of youth and a technology-based economy that's just beginning to thrive. It reminds me of this poem by...."

She grabbed my arm. "Did you get the mail?"

I hadn't. She sighed and got up without hitting pause.

The kids, I decided, wouldn't get off so easy.

"There are a few things you need to notice about this first episode," I told them a few days later when I'd managed to bribe them into submission. "First, notice the sense of loss permeating the narrative. Both Captain Avatar and Derek Wildstar are haunted by the loss of the son and brother, respectively. This is probably a reflection of the animators' own losses, or more likely, their parents' losses. If you look closely, the battle in which Derek's brother dies bears a striking resemblance to the final Japanese offensive at Iwo Jima, suggesting that...yes, Maggie?" Her hand was up.

"I want more popcorn."

"Just hold on," I said. She put her hand down and pouted. "Where was I? Oh yeah. The second thing you should look at is the soundtrack. While the theme song is reminiscent of a John Williams style fanfare, a style which actually dates back to the early adventure movies of the 1940s and the classic scores of Korngold, the Star Blazers theme actually predates Williams' most noted scores. Furthermore, the underscore of the show itself is a mix of disco and Wagnerian leit-motif. This is fascinating, because 1) it shows us just how much Western idioms had begun to influence Japanese art, and 2) it's all diatonic, which means that the composers rejected traditional Japanese harmonies in favor of a European chordal palette. This is especially important when considering the overall narrative similarities between Star Blazers and the Ring cycle. Specifically, the scene in which...Conor, where are you going?

He was hopping up and down, legs crossed. "I really have to go. Don't start it without me."

He ran off.

We waited. 45 minutes later I put in a Spongebob DVD for the little ones. By the time Conor emerged from the bathroom Maggie and Lucas were thoroughly engrossed in Spongebob and I was online, trying to find chat thread about non-linear narrative and its implications in post-colonial criticism.

It's a disease, I tell you.

So here's what I've learned about myself:

You know that dad who tries to live his football dreams through his kids? That's me. The mom who pushes her shy daughter into trying out for the cheerleading squad because that's what her mom did to her? Yeah, that's me, too.

I want so badly for my children to see the interconnections between art and culture. I want them to recognize that the creative act does not take place in a vacuum, that all art is bound by place and history, although great art transcends those bounds. I forgot, though, that what drew me to Star Blazers in the first place was the story. It's fun. Imagine that: a story can be fun. It doesn't have to be analyzed or dissected. It can just be enjoyed.

We did eventually watch Star Blazers. And you know what? They loved it. Absolutely loved it. Once their father stopped hitting pause to remind them that the clothes worn by the female characters both contributed to their objectification and served as emancipatory agents by freeing them from the bondage of 1950's gender roles, that is.

Incarnation, Part 1: Clocked

I love basketball. I'm not very good at it, but I love it nevertheless.

Part of me wants to take this opportunity to wax rhapsodic on the importance of seeking out opportunities to fail, and to fail spectacularly. But mostly, I just want to talk about scars and, more importantly, bodies.

I'm an academic. My days are spent lost between the covers of a book or poring over mountains of ungraded essays. I spend long hours debating the merits of Bahktin's heteroglossia or helping students recognize the difference between a iamb and an anapest. It's heady stuff, a cephalic feast, and yet at the end of the day I often have very little to show for my efforts other than the scraps and bones of half-gnawed metaphors and some stale prepositions. I live in the abstract, floating from idea to idea, and I've come to realize just how much I miss my body.

This is why I love basketball so much. I'm not good enough to play it with my head. For me, it's all elbows and sweat. I don't think (a truth to which I'm sure my teammates will attest). I'm the guy who remembers to pick but forgets to roll. No, I just go out there and run around. If they pass me the ball, I shoot. If not, I run around some more. Occasionally I'll set a screen on one of my own teammates. Like I said, I don't play with my head.

But that's why it's so cool that I got clocked. Loose ball on the floor. I dive. Other guy dives. Elbows and eybrows meet. Blood flows. He falls all over himself apologizing, and though I'm a little dazed, I'm just thinking how freaking awesome it is that I'm actually bleeding. I don't get to bleed nearly as often as I'd like, and now here it is, dripping on the freethrow line, leaving little red inkblots on the court. One of them looks like my mother. Someone drives me to the ER.

And so I'll have a scar, a little one, hiding in my eyebrow like a shy housecat. It's my red badge of courage, a tiny reminder of a life lived fully, at least temporarily.

Don't get me wrong. I'm not a sentimental advocate for "sucking the marrow out of life," a stance popular with Hollywood teachers and starry eyed English majors. Thoreau's injunction to live deliberately is sound, and yet it does not follow that all passions ought to be pursued with the same monomaniacial devotion. Nor do I endorse the testosterone fueled faith so disturbingly promoted in John Ethridge's Wild at Heart. If you haven't read it, don't. Ethridge attempts to make the case that American masuclinity is God's design for all men, that climbing mountains and shooting things and dragging women back to the cave by their hair (I am only slightly exaggerating) are all part and parcel of being made in God's image, that had Jesus been born today, he would have ended the sermon on the mount by BASE jumping off the top.

But this I do believe: our bodies are good. We're meant to use them. We're supposed to eat and drink. We're made to stand up and run and trip over things and skin our knees. This strange assembly of flesh and bone is made to jump and squat, to throw, to cry, to have sex, to type, to dance, to sleep. Though God gave us minds of incomprehensible depth and complexity, he also gave us bodies, bodies which are no less sacred than our most precious incorporeal creeds and beliefs, bodies which both house the soul and are the soul. The Platonic dualism that divides the body in two, condemning the flesh while exalting the spirit, is, for lack of a better word, dumb. Even Yoda, wise and wrinkled as he is, falls prey to this line of thought. In his desperate attempts explain the Force to Luke (who is, by the way, just about as obtuse as Jesus' disciples were), he finally grabs him by the arm and says "Luminous beings are we, not his crude matter." It's unsettling: nobody can mangle theology like a muppet can mangle theology.

Anyhow, although we proclaim the glory of God through what we think and say, the truth is that we could probably just shut up and let the wonder of the human body do the talking for us. Walt Whitman's ode to the human form, I Sing the Body Electric, confesses this to be true:

It is in his limbs and joints also, it is curiously in the joints of his hips and wrists;
It is in his walk, the carriage of his neck, the flex of his waist and knees—dress does not hide him;
The strong, sweet, supple quality he has, strikes through the cotton and flannel;
To see him pass conveys as much as the best poem, perhaps more;
You linger to see his back, and the back of his neck and shoulder-side... If any thing is sacred, the human body is sacred, And the glory and sweet of a man, is the token of manhood untainted; And in man or woman, a clean, strong, firm-fibred body, is beautiful as the most beautiful face....

My scar, therefore, is not a celebration of my hyper-masculinity. It doesn't make me bad ass or dangerous. It doesn't mean that I've lived deeply or that I've, heaven help us, seized the day. It just means that I lived deliberately and fully, with body and mind. And when I go to bed at night, I think about the science fiction that I love, about the projected futures in which we've evolved into pure energy, finally free from the shackles of flesh. And right before I drift away, I think scandalous thoughts, heretical thoughts, unforgivable thoughts. Just this once, I think, I hope they got it wrong.