LISTEN — Class Disrupted S4 E9: Shake-Up in the Assessment Market

Get stories like these delivered straight to your inbox. Sign up for The 74 Newsletter

Class Disrupted is a bi-weekly education podcast featuring author Michael Horn and Summit Public Schools’ Diane Tavenner in conversation with educators, school leaders, students and other members of school communities as they investigate the challenges facing the education system amid this pandemic — and where we should go from here. Find every episode by bookmarking our Class Disrupted page or subscribing on Apple Podcasts, Google Play or Stitcher (new episodes every other Tuesday).

In this episode of Class Disrupted,  Michael and Diane dig deep in analyzing the big acquisition of nonprofit testing provider NWEA in the assessment market by Houghton-Mifflin Harcourt, one of the largest curriculum players in the United States. They conclude that we should be skeptical that the acquisition will improve teaching and learning for students or that it will pay off as much as HMH might like. 

Listen to the episode below. A full transcript follows.

Diane Tavenner: Hey, Michael.

Horn: Hey, Diane. Wacky weather is continuing to roll out across the country it seems. And I hope speaking of that, you’ve all stayed dry enough in California, which frankly is not a sentiment that I normally need to express to you.

Tavenner: Well, Michael, the persistent rain has been a little bit strange, a little bit unfamiliar, but I will say, also welcome. Partly because we don’t have any personal complaints. We haven’t experienced significant destruction or risk and others have, so I want to be mindful of them.

But I will say, Michael, it does seem like our schools are never going to catch a break. Because with these storms have come power outages that have caused us to have to close several of our schools for several days, and honestly at this point we are kind of starting to wonder if we’re ever going to be able to consistently operate our schools with everyone in them.

Horn: It’s just insane to watch all this, Diane, and I’ve been following some of this with my friends in the Bay Area on Facebook and so forth.

But aside from weather and the normal question of our students in the schools themselves, which has been a recurring topic on a lot of our podcasts, I’ve been looking forward to talking with you because there was a major announcement in the world of assessments. And obviously assessments and accountability and the challenges they posed to the kinds of innovation that we both want to see are something you and I have discussed a lot on this show. But given the latest news, I’m just wondering if we can go much deeper on this topic.

I have a bunch of thoughts loaded up that I’m trying to sort out that are informed from the innovation side of my work on what this means for education. But I’ll be candid, I’m really excited to get on-the-ground perspective just to check my instincts on this.

Tavenner: Oh, you know how much I love geeking out on assessments. So I’m totally up for this. Before we dive in though, I’m pretty positive we’re thinking about the same story, but just to be sure. Are you talking about the one where Houghton Mifflin, or HMH, one of the three big textbook companies acquired NWEA, which as many of us know is a nonprofit that’s most known for what we call the MAP test. Which is basically an adaptive benchmark assessment and I think it’s used in about 10,000 school districts across the country and I will say personally I know used in a lot of charter schools.

Horn: Yep, that is the announcement that I’m talking about, Diana. It grabbed my attention.

Tavenner: All right. Well, mine too. I’m excited you’re bringing it up today. Because you know I’ve got a lot of feelings and thoughts about this topic in general and honestly this acquisition and so I’m excited to dive in. But it sounds like you have some real questions that we should unpack, so I’m curious about that. But before we do it, do you think I should give a bit of context and to help folks out?

Horn: Yeah, I think that makes sense, just so everyone’s on the same page before we dive in and geek out.

Tavenner: OK. Well, let me just, I’ll talk from my experience, and I was thinking, I think my first encounter with MAP, or NWEA, was probably around 2011-ish. If you remember, this is around the time of Summit’s redesign. It’s also a period of explosive growth for charter schools. And as a result, there were just a bunch of school leaders and philanthropists who were looking for ways to see if innovations, and I would say both charter innovations and school design innovations, were having an impact.

And interestingly, this is also the time period when Common Core is driving the full assessment conversation and what’s happening in the country. And so at least where we were in California, there was a bit of a space or pause on those big assessments while Common Core was coming into being. And so people were looking for things to fill the gaps during that time.

Anyhow, folks started using the MAP assessments. And a fun fact. Back in that timeframe, I remember giving the assessment to our 10th graders for the very first time and realizing that only a little over 2,000 10th graders in the entire country had taken that particular test.

I’m sure we’ll get into this as we start talking, but the number of people taking the test actually matters a lot given how it’s scored and what this test does. And certainly there were higher numbers in earlier grades. But the point is NWEA has grown pretty explosively over the last decade, at least from my perspective.

And so we fast forward to today, and NWEA is either a math or an ELA assessment. It’s administered three times a year — and I think that’s an important fact — by schools and teachers. It’s adaptive in that it’s on the computer and it feeds kids the next question based on how they did on the previous one.

And its purpose really is to show school year growth for our students and to give teachers and schools quote, “good information,” at the start of the year and the middle of the year about which students are on track to meet the grade level standards or to pass the end of the year assessment. And so conceptually, teachers and schools can make grouping decisions or curricular decisions or tutoring support decisions for students based upon that information.

But to be really clear, these aren’t really formative assessments that are embedded in curriculum itself. They’re not even tied to the curriculum. So they’re not driving instructional decisions and the absence of being connected to the curriculum…. Well, I’ll just say it as the company says it, is they really seek to be content-agnostic, and so there is a real distance there between what you’re doing in the classroom and these assessments.

Let me just give one additional piece of context, and that comes from the perspective of a parent. Because my child took these assessments three times a year for many, many years, and they don’t mean anything to students or families. I can’t sugarcoat it. There is literally nothing that happens for students or families as a result of these tests. The reports and results are basically nonsensical if you’re a student or a family and there’s just not any information in them that I can act on or use. And so in my experience, what happens is kids rightly start to ask the question of like, “Why am I taking this test so often? What is this for? What’s it doing?”

And as a school leader who was also that parent, I can tell you I spent a lot of my time trying to convince my own child and get him to convince his friends to do their very best on the test because it mattered for the school. And in reality it did because we were using those results for everything from charter renewal hearings to grant applications.

And so I’m going to stop there because that’s a lot, and you haven’t even introduced what you want to talk about yet, Michael. But as you can see, this is a juicy topic and I’m getting fired up already, so I’m super curious to hear what it’s provoked for you.

Horn: Well, I love this, Diane. I think it’s super useful context, and I love the passion and energy. I thought I would tap into it on your end, and so we have. But assessments more generally is something that I, as outside the classroom, as a non-educator, have really had to come up to speed over the last 15 years to better understand how different kinds of assessments can or can’t be useful. What their function is, what they are capable or incapable of telling you.

And I will say, I feel like I’m a constant work-in-progress on this, which is why I’m excited to check with you. But it is not really obvious to folks outside of the classrooms and schools oftentimes I think what an assessment says, and as a policymaker or someone like that making law about this stuff, it is very opaque, I suspect. That’s just further context.

But as I look at this acquisition, one of the stated reasons I think for it, and this is from the companies, not from me, but from the companies, I think the purpose of this assessment is that…. Look, you have this established assessment as a system that is used in lots of schools and districts across the country, and HMH now will basically be able to use those assessments to connect them with their curriculum and make personalized recommendations for each student.

The theory, I’m assuming, that they have at work is that they’ll be able to do what you suggest, or implied at least would be helpful. Which is that assessment should be used to actually be embedded in the curriculum and drive learning choices themselves, and in turn help bolster more personalized learning opportunities.

Now to be clear, they’ve been clear about this. Each set of products will still be sold separately and under their own brands. But I think a major rationale that they’re making is that the possibility now exists to combine these, to combine the assessments with the curriculum and buy them as a package, if you will.

Tavenner: I have to admit, Michael, I’m deeply skeptical. I’m really curious about the theory that suggests this is going to be a happy and productive coming together of these two very disparate things.

Horn: Yeah, well look, this is the theory or the theoretical reason for the rationale behind the acquisition. And why I assume as a nonprofit, NWEA probably said, “Look, this is for the good of our mission for the education sector.” I imagine that’s what they’re saying anyway.

But for the moment, I want to put aside one of the structural things that you mentioned that causes your skepticism, which is that these assessments aren’t, at present anyway, actually embedded in any curriculum or tied to any specific curriculum or given an on-demand way to drive student learning.

We’re definitely going to circle back to that. But I want to instead introduce a totally different theory on my end to analyze this move and its impact, and it’s called the theory of interdependence and modularity, and it’s one that we used in disrupting class as well as my most recent book, From Reopen to Reinvent, and even in parts of Blended. And it’s a theory that sort of combines engineering with business is the way I would think about it.

And as a refresher for those that don’t know and don’t geek out on the stuff routinely, the first half of the theory essentially says that when something is underperforming what customers or users need from it. And the way that two elements in the system interact are unpredictably interdependent, meaning the way one works and functions as dependent on the other one, the way it works and functions and vice versa. Then to make it good enough so that people will actually adopt it and use it, you have to do it in a proprietary structure and really have a proprietary design. And that’s because the two sides really need to be designed interactively.

So quick example from business, and then I’d love you to reflect on what that means for education. But way back when in the 1940s and ’50s when IBM came out with their mainframe computers, no one knew what a mainframe computer should look like or operator or anything. And so IBM just couldn’t exist as a standalone assembler of computers or even a seller.

They had to make every single part inside. The logic circuitry, the core memory, the operating system, assemble it, sell it, everything. Because each of those stages interacted in unpredictable ways with every other part of the process. And so if they had just thrown the operating system over the wall, if you will, to engineers on the other side assembling it, well, that would’ve impacted in unpredictable ways the performance of the fundamental computer and each side would’ve had to make unacceptable trade-offs. That just wouldn’t have been good enough and no one would’ve bought it basically, Diane.

Tavenner: OK, let me make sure I’m getting this before we go on, let me try to bring it into this context and see if I’m understanding it correctly. So in this case, we have curriculum, so textbooks, lessons, all of that, and a benchmark assessment, and they’re separate, they’re in separate organizations right now. But they’re interdependent because when you’re educating a child, you’re both assessing them and you’re doing this curriculum. Which is kind of confusing because curriculum has assessment in it as well, but we’ll leave that aside for a moment.

They’re interdependent, but they actually depend upon each other. But the key right now is it’s really unpredictable how that happens because, well, I guess I’ll just describe from my experience. Sometimes it often felt, like when we were giving NWEA, that the results of that were totally disconnected and unpredictable based on what we were seeing in classrooms and our knowledge of students and things like that. Is that, am I getting it?

Horn: Yeah, I think that’s exactly right, that there is certainly a dependence and in some systems it’s a well-understood dependence. It’s predictable. We can get into what that means later. But in many systems, it’s unpredictable.

You make a change to the curriculum, that should impact the assessments because otherwise you’re just not going to be getting reliable information and feedback on what a student actually has learned or knows now and can do with that information. And vice versa, if you make changes to the assessments, well that should mandate changes to the curriculum. And when they’re at arm’s length, they are not really feeding off each other.

And when you think about all the personalization that we want to see in the world, because students have different learning needs at different times, those interact in very unpredictable ways right now. Particularly as you get into, think about math or something like that and the sequence of learning and the dependence of different topics on other topics, etc., etc. These things weave together in very interdependent ways. They impact each other.

And so my sense is what you just said, which is that when you have the state of curriculum and assessment at arm’s length, separated from each other, they’re basically underperforming what schools need. Certainly Title 1 schools. I think you could make the argument in Lexington Public Schools where I am, that’s not the case. But in Title 1 schools in particular, those that serve lower income populations, for example, they really need these to be connected to help drive choices for the students on the ground, themselves.

And I think the evidence for this, Diane, actually, they don’t just have to take my word or maybe your word for this, it’s actually in the market itself. We’ve seen incredible traction and extraordinary growth of a different company over the past 15 years, which is this company called Curriculum Associates. And they make both the curriculum, the i-Ready curriculum, as well as the i-Ready assessments, and they use them as benchmark assessments, but they’re intricately tied to each other. And they essentially, in a closed-loop system, what you get on the assessment system determines what you do next in the curriculum and vice versa.

And interestingly enough, Curriculum Associates over the last 15 years has basically come from essentially nowhere to become one of the digital players that I would argue is disrupting much of the textbook market over that period of time. And I think it’s driven by the fact that their assessments and curriculum have been designed in an interdependent fashion.

And I think there’s further evidence for that in the struggles, at least relatively speaking, of a company called Renaissance Learning. And Renaissance, for those that don’t know, basically has historically offered the Star Assessments, another benchmark assessment instrument. And they’ve been sort of stuck, at least my sense is, with assessments where they’ve been trying to find partners, Diane, in the curricular space that they can align to, if you will, to provide information around.

But it’s been kind of kludgy because you’re integrating these curriculum that haven’t been built with the assessments in mind and vice versa, and you have a curriculum-agnostic assessment. And so they’ve acquired or partnered with providers like Nearpod or more to essentially have both sides of the equation.

But because they weren’t designed interactively, the feedback loop between them, or at least the theory would suggest … And full disclosure, I can’t make the judgments because I don’t use them. But my sense is that the theory suggests that the recommendations and integration of the two just wouldn’t be near as eloquent, if you will, as what Curriculum Associates offers because of the way they’ve built both sides of those items. And so to be clear, that means in a not good enough world where these things are underperforming, that favors interdependence.

Tavenner: Fascinating. What you’re sharing is bringing up two things from me, and so I want to just keep checking my understanding of this with you.

The first one is, it seems to me that maybe another example of this is the partnership between the College Board and Khan Academy. As folks began sort of pushing back on the SAT test as sort of these standalone assessments that are super high-stakes and they just felt like they’re disconnected and inequitable, the College Board partnered with Khan Academy to offer the learning curriculum part of the equation. The benefit there was it’s free, it’s to anyone who wants it. Bringing the equity piece. Is that a good example of this theory?

Horn: Yeah, and I think we’ll get into more how it is in a moment. But it’s sort of right, it’s this modular trying to build up into an integrated way, I think.

Tavenner: The second thing that comes up to me, which probably you’re like, “Of course it comes up to you,” is just what we did at Summit Learning. And it seems as you’re describing the theory, that this is what we did intuitively. We built, as you know, an entire system of learning and assessment to be interdependent and integrated from the very start. And we just thought it was the only thing that made sense. And I know we’re not alone in this realm, but there’s a reason that a lot of people don’t do it. There’s a whole bunch of reasons. That’s probably a whole other episode.

But it seems you’re offering evidence for those instincts. But I’m wondering about the modularity part of it, because we haven’t talked about that yet, and so how does it fit in?

Horn: Yeah, so I’m glad you asked because there’s a trade-off in this and there’s not one right place to be in this continuum. Because when you’re optimizing for raw performance by creating a really proprietary interdependent system, the trade-off is that you can’t get affordable customization. And this is a really important point, and it’s interesting actually because Summit Learning, you optimized for curricular modularity to create an interdependent system around parts that were underperforming, and so you made these sort of trade-offs.

And that’s the other piece of this, is that in practice, every system has some modular components and some interdependent ones, and you basically want to be interdependent where the performance isn’t good enough and modular where you’re overshooting so you can get that customization. And so in essence, what the theory says is that when you start to overshoot, that’s where you need that customization because people demand it, they want affordable customization. They want to be able to mix and match different parts and just pop them in, and to do that, you need modularity.

So the quick example, again, we’ll go to computers, is Dell computers, say, in the early 1990s. If folks remember, Apple, IBM, they had started to over-serve what people needed. They weren’t willing to pay for some of the improvements in that paradigm. And here comes Dell and they don’t make any of the parts inside of the computer and that means you can just jump on their website and be like, “I’d like this much memory, I’d like this kind of drive from Seagate, I’d like this kind of monitor, etc., etc.” And they fit together in very well understood ways and so Dell just quickly snaps the parts together and 24 hours later, you got shipped out an affordable customized computer.

And so I think you at Summit Learning know these desires very well. Because you chose to avoid purchasing on the open market a proprietary highly interdependent curriculum, and instead, as I said earlier, create something far more modular on that side because you wanted customized playlists and so forth for the students.

And interestingly enough — I’ll try to make the argument for you — I think you said that where the systems out there are really underperforming is in the integration or the interdependence, if you will, of the knowledge, skills and habits of success that are often broken apart and perhaps unfortunately so. And you said that’s where we really need the interdependence in the Summit Learning system.

Tavenner: I think that would be… 

Horn: Yeah, go ahead.

Tavenner: I think it’s all just clicking for me as you’re talking. We’ve never wanted or bought a textbook, because it’s so not customizable, and if you’ve ever been a teacher or know a teacher, you know that is the number one thing a teacher wants to do and needs to do, which is why those products don’t get used with fidelity. And so we are just, I think as practitioners so keenly aware of what actually happens on the ground and pragmatic about that, that that’s where we wanted customization. But super tight on the assessment part of it and the connection to it. And like you said, the connection of all the factors of learning, not just a single siloed content area, if you will.

Horn: Yeah, and I think that’s the big aha here, which is that you can’t actually just jump into a modular design. Everyone wants customization, it sounds great, but you can only move to it once the interfaces between the different components are predictable, specifiable and verifiable.

And I hear this all the time, Diane, in education circles, that we want an open system, we want an open this, we want an open that. As though open, in and of itself, is a good thing. And the only way though you can move to that kind of modularity is you really have to understand how each side works and functions and the impact of changes on one side with what it’ll do to the other.

And that means you have to be at the level of predictable causality. You need to be able to create specs and essence for exactly how those parts fit together at the interfaces. You need to be able to verify that the parts in fact meet the specs. And then in the early stages of creating something new, when it underperforms, that’s just really hard to do and it’s only when you get better and better and better, you start to over-serve people and you start to understand how these parts work.

Then you can start to shift to modularity because you have a better sense of how the overall system is interacting and the science of studying them essentially allows you to get more predictable. So a decade hence, maybe we could start to modularize Summit Learning, but we’re not there yet.

Tavenner: I’m having a reaction that I commonly have when we are talking, putting theory to practice, and often in my mind, I’m always putting it to our practice. And at first when you are describing the theory, my mind immediately goes to, oh no, we didn’t do that. We need to go back and do that. Oh no, we messed that up. We weren’t good innovators.

And then it’s funny, as we talk more and more, and I really calm myself down and think about it, I realize that a lot of what you’re saying, Michael, we do or did and that the theory really describes our behaviors. Which I think is good because you don’t want a theory that’s actually not describing what’s happening.

And I’ll be honest, the outcome of that work is strong. By designing a learning and assessment system from scratch that clearly prioritizes each and every student and incorporates the learning science and the full understanding of the needs and leverages technology, we really have created what I think is a very elegant and interdependent curriculum and assessment system that is highly modularized in the right places and thus customizable for the folks on the ground. Both students and teachers I would argue, and schools for that matter.

And as I say that, I’m cognizant that it sounds a little bit like bragging. But the point that I want to make is that it’s possible that what you and I are talking about, it’s not a fantasy, it’s not a future thing. It literally is happening right now in hundreds of schools across the country. And I just think that’s important because we always want to take this sort of third way future-looking perspective and what I don’t want this episode to sound like is just us criticizing a deal that happened without another solution. Because I do think there are other solutions and it doesn’t feel like we’ve compromised in the combined offering. And honestly, this acquisition feels like a compromise to me.

I suspect that a lot of people who care about seeing big rolled-up data and tracking and reporting on the percentage of kids in schools who are supposedly learning or not, they might be happy with this direction, Michael.

But for those of us who care about literally every single child and if they can read or solve problems and if every child’s motivated to come to school and knows how to learn, for those of us who think it’s unacceptable that we’re… School systems are unable to simply tell families if their child’s on track to be a skilled adult in areas that families care about in their life goals, and instead we send them these incomprehensible test scores in the mail months after a child took the test.

Well, look, I’m going to stop there because you could see I’m getting fired up again. But this is possible what we’re talking about, the theory says we should be doing, and I don’t think that this deal represents that.

Horn: And this is where I ultimately want to go as well, Diane. And I think there’s another issue as well underlying this, which is, it gets the multiple purposes that assessment serves. From driving learning to continuous improvement to accountability and transparency. And I think what’s also interesting on this is that the interdependent approach, frankly, it’s always going to struggle to serve a public accountability function in my mind in which we’re making judgements about the schools themselves, even as the theory shows why it’s the best way to optimize the teaching and learning.

And I’ll say that, again, to analyze Curriculum Associates for example, and with immense respect for the predictability of i-Ready assessments on what a student will score in a summative year-end assessment. But essentially, Diane, if the i-Ready assessments… I suspect some people are listening to us are just like, “Well, why wouldn’t we just use the i-Ready assessments to replace the summative year-end assessments. They’re curricular tied.”

Great end of story. But you can imagine that then there would be a really slippery slope that would come into play in which Curriculum Associates — And again, I’m not saying that they do this nor that these incentives exist right now, but in this alternative system I just described —  the incentive I think would exist for them to basically just give rosy assessments to student learning to show, “Hey, look, our i-Ready curriculum is really making a positive difference for students.” Because there’d be no check on the other end of it.

And then if they made those changes, which are no longer reflective of actual performance, now that means that the assessments would no longer be as useful for guiding learning for students and teachers. There’s a big trust issue here, in other words,

Tavenner: Michael, I experienced this in real life, where very few people actually take seriously the assessments built into Summit Learning as an outside valid way of measuring what we’re doing. I argue that they should because they can be completely valid and reliable in all of the things that we need. But there’s very little pickup there, and I suspect what’s underneath it is this trust issue that you’re surfacing right now.

Horn: Yeah, well, so I think this is where there’s another approach from the theory, which works frankly though only in a world of mastery learning that we’ve discussed in the past on this show in which students work toward mastery of each and every competency and time becomes the variable, not the student’s learning as the present system holds.

But that is if you have the interfaces clearly specified between curriculum and assessment, and that’s a big if, then you could have modular assessments created by a third party. So not the curriculum company itself. That an individual school or a district, or yes, even a state as Louisiana has, could select based on its alignment to their curriculum. And in that world, those assessments could both serve to inform instruction. They’re learning informative, but they’re also summative because the measures they’re giving are inherently of learning. Which is what your system does, just, to your point, it’s not third party, so people don’t trust it.

And the point being in that world, if I demonstrate mastery, I move on and that can be reflected and it’s more robust than a summative assessment because I can demonstrate it when I as a learner am ready to raise my hand in show mastery.

And so to come back to NWEA, this is the curve ball it seems to me that HMH is walking into here, and you’ve alluded to it several times. But just to say it from the theory perspective. On the one hand, by controlling the assessments, they theoretically, theoretically, different from theory, will have the opportunity to make them in a proprietary fashion alongside their curriculum and benefit from the same way Curriculum Associates has. But to do that, they have to do it from starting with assessments that are not curriculum aligned. That’s going to be a major lift.

And on the other hand, HMH can’t exactly serve the modular use case that I just outlined because it doesn’t have an array of assessments aligned to lots of different curricula that can be customized where the individual school or district or state or charter organization can select to drive learning. And it now owns both sides of the equation. So it’s sort of caught here, Diane.

Now I think there’s some silver lining here as well. There’s New Meridian, for example, a nonprofit. It exists as a newer third party provider seeking to offer modular baskets of assessments that you can pull off to match your curriculum. But I’m just not sure that HMH can make this jump as we’ve outlined.

So it’s possible, Diane, that it’ll be commercially successful as a sort of in-between solution, I guess in the market as the market maybe transitions over time. But I’m frankly not sure that that’s where the puck is going or frankly where it should go. And I really want to know how that lands on the ground for you.

Tavenner: Well, you probably have guessed that the idea of a commercially successful in-between solution isn’t landing very well with me. This is the thing that just really frustrates me about our sector and our work.

I agree with your assessment, Michael, and I think this is another example of what holds us back in education at the broader scale. We’ve got these two players who have a massive market share together. Well, I would call it massive, you might correct that, but it seems pretty big. And schools have to have curriculum and assessments and they’re under pressure to have something that tells them exactly where students are, especially coming out of COVID, so that they can help them direct resources to them. And just the expectations on teachers in schools continue to go through the roof around what they are expected to do.

And so tons of schools are going to use this joint offering, and in their mind they’re going to be told and sold on the idea that it’s better somehow and it’s going to suck for kids. I don’t know how else to say it.

It’s not better. It’s not good, and it’s not what our kids deserve and what’s possible. And what’s frustrating about that is it pulls the vision, the mind share, the resources, the energy away from what doesn’t suck, which is what we were describing before. Assessments that are embedded in project-based curriculum that’s authentically assessed in a way that dramatically improves engagement and learning and self-direction for students and job satisfaction and impact for teachers and gives them local control and all of the things that we care about that we know matter. And we have the capability to do that today, do something significantly more meaningful. And I just think that this pulls us away from that work. It pulls everything away from that work.

And I think it goes back to the conversation we had last episode, Michael, when we acknowledge that most people don’t even think about students as customers. And in my mind, this approach sort of epitomizes that. As I read the transcript of the interview with the two CEOs, there was literally not a single mention of this acquisition in terms of how it was valuable to students. They got down to teachers and said, “Teachers would have all this data and information.” Of course, the way I read that was like, “Wow, teachers have now a new expanded job.” And having had that information, know how hard it is to actually work with it.

So yeah, I’m going to stop again because I’m… But that’s how it’s landing with me honestly.

Horn: Well, no, I think that makes sense. But something you just said in terms of all the information we’re going to get and all this stuff and so forth. Let me step out of my innovator hat on that for a moment and talk about one other problem, in my mind at least, with NWEA assessments more broadly through an explicitly education lens. And this surrounds how so many of us want to shift to measuring student growth instead of point in time learning.

But the reality is that there are different ways to measure growth. And the public, and educators I think honestly, broadly speaking, don’t understand the difference. So when they see a growth measure, they don’t actually know what it’s telling them. And in this case, NWEA reports a norm reference growth measure as opposed to what would be called a criterion reference measure.

Essentially what that means is a norm reference basically evaluates you relative to other students like you at your “percentile,” quote unquote, of your learning. Whereas a criterion reference is against that yardstick of curriculum, but it has to be curricular aligned to be able to do that.

Now, as a result of that, frankly, the NWEA growth measure offers this incredibly false illusion where it basically, if I’m, say, a student in the fifth percentile of seventh grade or whatever else, then basically it’s going to compare my growth relative to other students in that same segment. And so I might grow one and a half years relative to those students. But that’s embedding all the lowered expectations and malaise of our education system in that growth measure.

When a parent hears one and a half years of growth, that’s not what they’re thinking. They’re thinking, “Oh, my kid grew from say the third grade to well into the fourth grade.” That’s what policymakers, I think here, that’s what I think educators and curriculum companies that use this stuff, I think that’s what they’re hearing too, and it’s just not what it’s measuring.

Plus, I think this gets back to what you said at the very beginning, and maybe this will start to wrap a bow around this. You mentioned that when you gave it to your 10th graders, only 2,000 kids in the country had used that. So now it’s driving a growth number relative to other students like them. I’m not going to be able to do the math that quickly in my head, but like 100 percentile points, whatever, it’s not a lot of kids in each percentile. You’re driving a growth measure off of that. That’s even less meaningful, I think, Diane.

Tavenner: Well, you’re opening the can of worms, and I know at this point we should be closing it, so I’ll be brief here. But I distinctly remember that conversation about those 2,000 kids. It stands in my mind a decade later because it was so clear to me, not initially, but when we really, really, really dug in to understand it, when the company rightfully said, “Well, there really aren’t kids like a lot of your kids.”

So you have to take these results with a grain of salt because there’s only 2,000 kids. And, I mean, Michael, we could do days on this, but let’s just start with the concept of, yeah, you’ve reinforced why these results are meaningless to parents and students because how can anyone in their right mind understand what you just said when you’re looking at a result or expect that that’s what you’re getting? You would just want something straightforward. Like, do I know this? Do I not? Am I good at this? Am I not? Yeah. Anyway.

But the second piece is this concept of bias. So we’re going to just compare you to people who are like at that moment in time, if you’re a quote “lower” performing student, what are the expectations? We know what our systems have in terms of expectations. We know what natural human inclination is.

I just think this is so destructive, and I’ll just share one last story about this. Part of those conversations with my son around this were so disturbing because he would come home and say, “The kids would just sort of share their raw score with each other and they would just stack rank the scores.” And what happened every time is quote the “smart kid, the super smart kid,” always got the top score.

So in my world where I’m caring about growth mindset and kids believing that hard work actually pays off, I was having to fight against what these results, the messages we’re sharing with kids, which was much more of a fixed mindset than a growth mindset. And we were doing battle against that in the context of the school because we were giving these tests, which I see the look on your face, and it just is like, it’s icky. There’s no other way around it.

Horn: It’s really crummy. It’s really crummy. All right, well, let’s not leave it there because I think there is hope on the horizon. And maybe Diane, frankly, as we see this acquisition, maybe it represents consolidation and often when you see consolidation in markets, it creates room for the disruptive plays underneath to gain market share. And so I’m hopeful that we will move to content aligned assessments and have some third party players working hard to align to the different curriculums out there so you can select the right ones for you and move to this world where it’s both driving learning, but also giving parents and students the information that they want about their growth. So I’m hopeful we’ll get there, but I don’t think this acquisition is the step.

Tavenner: A perfect place to leave it, Michael. And before we sign off, I would love to hear what you’re reading, listening to, or thinking about outside of our day-to-day? Add to my list?

Horn: Yeah, you bet. Well, so Sal Khan sends a book every few years it seems to folks who are on his advisory board. And so I got the one in the mail from him, which is The Art of Living: The Classical Manual on Virtue, Happiness and Effectiveness by Epictetus. And the interpretation that he sent is by Sharon LaBelle. And I read it chunked, like a few lessons at a time over the last several weeks. And it was really helpful, Diane, at just resetting my own sense of how I think about my own life lived and what we can control and focusing on that. And so I really enjoyed it and was appreciative that Sal made the time to send it. What’s on yours?

Tavenner: Well, that’s awesome. I appreciate that. I have a fun one this week. So as we’re about to head to India, folks have been hearing about that for a long time. We watched a movie that’s on Netflix, it’s called RRR. It’s getting some positive press in the U.S. It is a Bollywood movie out of India. The hard part is it’s three hours, but I will say it is a fun three hours. It was joyful and interesting. It’s packed with all sorts of over the top action and it is over the top in many ways. But we deeply enjoyed it. And I think on the sort of interesting part of it, potentially quite controversial, but it really is offering a different narrative about the drive in India to be their own independent nation and how they engaged and interfaced with the British. And the normal narrative we all have is surrounded by Gandhi and a peaceful approach and there’s some folks who want to have a more fight struggle narrative. And so that is presented here for better or worse.

Horn: Super interesting. That’s a major political current right there, I know. And so I can’t wait to hear what you’ll learn from it as you come back if you choose to report on that. But for now, we’ll leave it there. Thanks for engaging with me on this conversation. And for all of you listening, we’ll see you next time on Class Disrupted.

Michael B. Horn strives to create a world in which all individuals can build their passions and fulfill their potential through his writing, speaking, and work with a portfolio of education organizations. He is the author of several books, including the award-winning Disrupting Class: How Disruptive Innovation Will Change the Way the World Learns and the recently released From Reopen to Reinvent: (Re)creating School for Every Child. He is also the cofounder of the Clayton Christensen Institute, a nonprofit think tank.

Diane Tavenner is CEO of Summit Public Schools and co-founder of the Summit Learning Program. She is a life-long educator, innovator, and the author of Prepared: What Kids Need for a Fulfilled Life.

Get stories like these delivered straight to your inbox. Sign up for The 74 Newsletter