John Hattie: “I’ve got, it must be the day of fours, I’ve got four aspects that I want to talk about too in terms of the notion of professional learning, about the Charter (PDF), about how I read some of the research in professional learning relating to it, and I obviously want to put a challenge to AITSL and this group.
“Standards without an understanding of their impact and their assessment have zero effect” (John Hattie)
Let me start. When I first saw the Charter, the standards, it was kind of like ‘here we go again’. I went to Google last night and googled charters for professional … , I see the most modern part of this is we’ve changed the title from professional development to professional learning, hundreds of them. And I want to relate back to a couple of experiences in my career.
When I first went to work at North Carolina, I had the job with the National Board for Professional Teaching Standards and the first job they gave me was to get together all the professional standards there were for teachers right around America, across the States, within jurisdictions. I literally ended up with a room of boxes. My job was to review them, the bottom line is simple: they had absolutely zero effect.Another job I had as the psychometric adviser to NCATE, National Council for the Accreditation of Teacher Education, and they have quite a few thousand education institutions they accredit, and at the time in the late 90’s they were very nervous because many of the teachers that had passed those standards were failing in the schools.And so, what I learned from that experience was, and my major message about this document here [the Charter (PDF)], is standards without an understanding of their impact and their assessment and their evaluation are, quite frankly, only worth the paper they’re written on. And so my major challenge to AITSL moving forward is how do you now come up with evidence about the success of the implementation of these standards.
“We love to study failure. There is an incredible amount of success.”
One of my worries in academia and in the business of education, is we love to study failure. There is an incredible amount of success out there in our systems and I would hope that we have an opportunity with these standards to look at where success is, articulate success, welcome it, embrace it and demand it of the system.We know a tremendous amount about professional learning. In fact it seems strange to me at times that some of our adults in the professional learning seem to think ‘not for them’. That is the very thing we ask kids to do, to come to an institution that is deliberately set out with a focus on learning, with a curriculum, with an assessment, with a whole set of understandings about pedagogy, and participate in that and because of that they’re going to be better citizens when they leave us around 15, 16, 17.So it seems a mystery at times that we then seem to think that it doesn’t apply to us. What applies to us is osmosis – we will learn, just leave us alone. This is the business we’re in. And so I have no difficulties with trying to get a better understanding about the learning of the adults in our education system in the same way that we’ve spent aeons looking at the education of the students. It shouldn’t be difficult, we almost shouldn’t have a document like this, even as I understand why we do.Where I want to start though, and is leading from what Michael said, my view is that if you don’t start respecting the teachers where they are, professional learning has no import. And if I know anything, teachers have incredibly strong theories and if you don’t start by trying to understand the theories that the teachers bring to the professional learning, you’re walking up hill the whole way.
“There is no difference between boys and girls in terms of achievement”
Now, in many cases, those theories I have no difficulty with – my point is that we need to know them. I do get very frustrated when I hear teachers’ theories about, for instance the latest mind-storm that has gone through New Zealand is some idiot woke up one morning and said ‘Ah, I’ve solved it. There’s only two ways people think – a boy way and a girl way.’ And we’ve got this whole rhetoric about boys education. And when I look at the work Janet Hyde’s done, the work that I’ve done when I’ve got studies of many millions of kids the question does it make a difference, there is no difference between boys and girls in Australia in terms of achievement, so why are we inventing theories about why they’re different?We have a whole rhetoric about discovery learning, constructivism, about learning styles that has got zero evidence for them anywhere. And my point being that if you don’t start by acknowledging that teachers come with those types of theories, you are going to wonder what they’re based on. And one of the things that strikes me as I do my work on Visible Learning is, like you kind of think as the arrogant academic, maybe I kind of do know some answers so why don’t they share these. And the reason is simple, I haven’t got enough listening and respecting skill to start by understanding where the people are coming from.And it isn’t about the learning styles nonsense, it doesn’t take long to think wouldn’t it be more powerful for you as the teacher had multiple styles of teaching, and you could reap all the benefits of how you believe about that. And you can get quite an impact from that, and so where do you start from this notion of teachers having war stories. And I don’t mean that in a negative sense, they have a very strong set of war stories and so when I come along with the evidence I have I know its going to crunch.The same in the debate about professional learning. When you go into a school and say we’re going to have professional learning and we’re now going to have these kind of impacts, you need to understand where they’re coming from. And as I go around schools and see their lists of professional learning, many are exciting and many of them make me sick. Where are they starting from?
“0.4 is written on my forehead”
Let me go a next step and that is if, the one thing I’ve found from all the work I’ve done surprised the heck out of me and now drives me, is that in education everything works. And so I have no time for the concept of what works because the answer is everything.And just a couple of slides to put you in the picture for particularly my later comments, what I’ve been doing is that I’m now up to 900 meta analyses around 260 million kids, try to take all the things I can to put along this continuum to try to understand what’s up here and what’s down there. And the average effect across them all is around about 0.4. And so what drives me, for those who know me and work with me, is 0.4 is written on my forehead here because, on average, that’s the kind of effect size you get.Now if you don’t understand effect sizes and Z-scores and all those things it doesn’t matter, it is the comparison. Because here’s 240 million kids and the point I want to make, the red zone that’s zero and below. That is where there is something that happens that actually decreases achievement. There’s virtually nothing. And there are some of those that are very sensible. The effect of one or two disruptive kids on the achievement of all the kids in the class is -0.3. And when you take those out 95-98% of the things that we do to people to enhance their learning works.
“If it doesn’t have an impact greater than the average, we’re doing the wrong thing.”
And so, you know the story, all you need to enhance kids’ achievement is a pulse. I think that’s the thing that we don’t realise in this business. It turns out that it’s not a bad normative distribution, centred around the 0.4. What I want to study in my life is what is the understanding, what is going on, for that group of teachers, which is about 50-60% if you look at it from that distribution that systematically get effect sizes greater than the average, compared to those that get it less than the average?Whereas 99% of our debates in education, particularly at systems levels is how do we enhance achievement. Wrong question. On the other hand it’s the right question if you like it that way. Because anything you say will work. One of our three ministers for education in Victoria goes to bed last night wakes up this morning with a bright idea – it will work! So how do we strive up there? So kind of Tony, Margery, you know where I’m going. If the professional learning does not have an impact greater than the average, we’re doing the wrong thing. And so how can we constantly ask what is it that gets us to success?
“I am interested in the impact of teaching.”
Just two slides to give you the extremes. The disasters, the things that hopelessly don’t work and I’m echoing the point, and that’s the trouble with following Michael Fullan – he says everything, echoing the point the majority of the things that we love to debate, not only at the political level but also in the staff rooms and amongst the parents, are the structural things. The things that you can actually see and do and touch and legislate. They don’t work, they’re well less than 0.4.All that, well you can see them there – I’ve just given you a snippet and a sample, I try to suggest to you that they will dominate the debates. We love to have programs of professional learning that, I suggest to you, about things that don’t change us. We don’t mind change, as long as you don’t change me. But if I go to the other end of the scale, you get the stuff that works and its heavily embedded in that third stool [re: Fullan presentation]. And where I’m at in my own thinking is that I’m not actually interested anymore in teaching, I’m interested in the impact of teaching. I don’t want to have a debate about direct instruction or whether you should use reciprocal teaching and all those kinds of things, I care about the impact of those things.Because what I find across the whole range of effect sizes is teacher effects everywhere below and above 0.4, so it’s not true that teachers make the difference – it’s teachers thinking certain ways and it’s the thinking that matters. And when I try and sum up everything I know in this business, it’s that and you can wait for my new book in December or you can get Michael’s one that came out a couple of months ago that had a chapter “Know thy impact”. Because as far as I’m concerned, what drives me is those teachers those systems, those school leaders that understand their impact. The concept is the teacher is the evaluator.
“The most bankrupt institution I know in this business is teacher education.”
In the same way that I want to argue to you this afternoon, that we should be asking about professional development in terms of the evaluation of its impact subsequently on the students. Too often we ask about the impact of the professional development on the teacher. I want to go the next step. Just to make sure I’m an equal opportunity critic this afternoon, I also want to criticize the most bankrupt institution I know in this business, teacher education. And the reason I think they’re bankrupt is that they’re constantly solving the wrong problem which is how do we improve the quality of the teachers entering the teaching force.The right problem to me, which is one of the reasons I came to Melbourne that is one of the few that have actually tried to do this, is: What is the impact we’re having on the students of our student teachers? And you do totally different things. And that’s the kind of message I want to get across this afternoon with respect to the whole thing about professional learning. It isn’t about professional learning, it’s about the impact of that professional learning on the students. Now, hey, it’s not an easy connection, theories of action are lovely things. But there needs to be a corpus of evidence that relates to that impact beyond the satisfaction of the teachers.
“One policy doesn’t fit all”
And the last of my four things here is that the worry I have, is that too often the way it is for teachers and school leaders is that professionalism is defined by their autonomy, in the same way schools like this notion of autonomy. Here’s my simple answer – nobody has the right to go into a school and systematically get effect sizes of less than 0.4. And one of my troubles with most policies is that they’re invented to solve the bottom of the problem and then they’re applied to the top half of the distribution and then we wonder why they don’t work.Linda Bendikson for instance has been working on the nature of principals in struggling schools in comparison to very successful schools. In struggling schools you need quite direct leadership, in non-struggling schools, indirect leadership works. So one policy doesn’t fit all. I’d have a strong hunch that if you go back and look at No Child Left Behind it works very well in those States that were below the average but for those States above the average it probably has zero to nothing impact in terms of achievement. It might have an incredible impact on frustration and the deviation of attention and resources to the wrong thing.So my worry is that we also have to be very careful when we prescribe this kind of notion [the Charter] that we’re not trying to fix the bottom and therefore have no effect on the top. I care about the top and I care about getting more people in that group called the top. So where does professional learning fit all in this? Well, I think it’s about these two things – The second one is about the impact of the character of both the teachers and the students. So then let me come specifically to the Charter.
“How do we get Professional Learning more sustained?”
The first thing is there is, and I’m delighted to see this and I have to say Margery that it wasn’t there in the first version and I appreciate that others made this comment, not just me, but there is a beginning of an emphasis in here on the impact of professional learning on these kinds of outcomes. And as Lawrence Ingvarson and the ACER team have long argued getting away from evaluating professional learning in terms of whether the tea was warm or the seats were hard. Can we evaluate it in terms of that kind of impact?I’m not sure about this autonomy, one. Number two, I have this hunch that too often some schools choose professional development learning as they choose their movies. For a particular time, make it entertaining, make me think but when I go home, doesn’t matter. How do we get that more sustained? And I’m going to challenge you [AITSL] to come up over the next year with some evidence from our communities about professional learning that has had an impact on student learning, to start to drive the debate about what effective PD looks like, not just, I’m sorry, all PD. I’ll come back to that one.
“Teachers’ subject matter knowledge counts for zero”
There’s nothing in here [the Charter (PDF)] about content knowledge. Now the scariest thing I know from the Visible Learning work I’ve done is that teachers’ subject matter knowledge counts for zero. It’s a 0.09 effect, and that scares me and that says we’re doing something drastically wrong. Now you could tell me that it should and I know it should. But how I’ve looked at it and Mary Kennedy has done the synthesis of the qualitative literature, it doesn’t. It doesn’t in here [the Charter]. Maybe I know why – we’ve got to change that and we’ve got to change and have a better understanding of why subject matter knowledge is not accounted up to now in terms of its impact on kids.I love the move to collaborative teams, professional learning communities, those kinds of things provided they get beyond building teams. One of my major arguments in this work I’m doing at the moment is how we can get teachers more involved as they plan lessons to work together to do it. One of the beauties of our profession is that we’re the best critics of each other in the world. So how do we optimise that power so that we criticise before we teach.
“How do we get the success criteria up front?”
And the other thing I like to do with the work I do with teachers is to get them to construct the assessment they’re going to give at the end when they start. Because that clearly articulates to them what success looks like. Because so often when you get to the end of the lesson you create an assessment about what you’ve taught as opposed to the concepts perhaps you wished to teach.I think the same kind of notion here, I’d be saying to communities going to professional learning what is it you want to change, tell me up front. And there are a multitude of these I’m not suggesting just one and I’m not suggesting just test scores. But how do we get that success criteria up front so that collaborative teams can have a focus to which they move to. The focus isn’t what was delivered, the focus is the impact it had subsequently back in the school.
“How can we get common understandings of progression and challenge in our schools?”
A lot of the work I’ve been doing in New Zealand over the last 12 years has been this notion of overall teacher judgements. And like all good concepts the OTJ acronym is now right throughout the school system and in fact New Zealand is going to an election in the next three weeks and OTJ is a very big part of it.We asked the teachers to defend their judgements in maths, science, phys ed you name it. And we say to them if you used a test you failed, if you don’t use a test you failed. Ultimately we want to know what changes, what we want to look at is about how you make judgements about kids and their performance. Now a lot of our teachers dislike this intensely. And so when you talk before about how you make the evidence public, this is one of the things that I want to work on too, is how do you make teachers’ judgements explicit?When I work with schools, including some here in Victoria, there’s some stunning stuff going on with teachers who can defend their judgement but when they do it in groups the discussion often closes down very quickly because that’s your judgement, this is mine. How do we collectively get a concept of progression? Because if we don’t it’s too much of a random crap shoot every time a kid touches a new teacher.And I think this is one of the things that professional learning can have a dramatic effect on: it’s how we can get common understandings of progression and challenge in our schools so that when kids do go up the grades, when they do go up the nature of the work that they’re doing, they are increasingly challenged by teachers that have common understandings of progress.I have to ask what’s excluded, it’s always my test in any document and I’m afraid I don’t see anything excluded. I want something excluded, it is just not true that everything works. That is one of your next jobs.
“You either don’t get employed again or we don’t pay you.”
One of the things that I’ve been doing in this area and again, I don’t want to keep going back to New Zealand but it was my last 12 years, is kind of a laugh you can do this in a small country with various ministers and have been the adviser of 7 of the last 8 – one only lasted 6 weeks so I missed that one, and one of the things that I’m saying to them constantly is that in the professional development area one of your concerns is your model of doing it in terms of who you employ.I’m also on the Board of a company that has a large role in the education in Abu Dhabi, and they’ve gone through so many consultants it’s incredible, with very limited effect as it turns out. And what I convinced the New Zealand Government or the Minister to consider is start to put some onus on the professional development people and say to them that if you don’t get with the school you’re working with the impacts that you’re looking for, you either don’t get employed again or we don’t pay you.In Abu Dhabi we put $20million up for grabs, we have 30 schools there, and the argument was that if you don’t get these effects in these schools over the year as the professional development providers with the schools, you don’t get paid. We got 29 of our 30 schools through. There was an incredible attention to having an impact. This is what the New Zealand Government now says to PD providers.They’ve done a lot of work looking at the kind of growth change (and I’m starting to look at the NAPLAN stuff, thank you Peter, and the kind of growth change and the good news there is that the growth change is reasonably well correlated with the socio-economic indicator of the schools, you can get growth right across our system) and saying what kind of growth changes and in the New Zealand system it really is heels to the fire for PD providers.I would hate this [the Charter (PDF)] to be thought of as something where PD providers can go mad. They are implicated in having success and I’d be asking you to think seriously about that so that we are all working together and not when I go in and do PD and nothing happens I’m going to say ‘it was them’. We are good at that.
“Don’t ask for the data. Ask for the evidence.”
So let me look for a moment then at this notion of how I see the research on PD. And I’m putting this up to make sure that no one leaves this room thinking that I’m asking for just narrow test scores. As we did with the NCATE, we said to the teacher education institutions we’re going to: Ask you two questions – what evidence can you show us that you’re having an impact? And what are you doing about it? Now, they disliked it intensely and said ‘no, you just tell us’ and we said no.Now what we had to do next, Tony Margery what I’m suggesting does next, is you come up with examples of the kind of evidence you would accept because we don’t want to constrain the schools to all using the same methods, because its just not true out there that they all have the same problems and issues and directions.And throughout the last 15 years of my career as I’ve developed the New Zealand assessment system and that those two questions have driven me and I’ve actually convinced the education review office, inspectorate office when they go into schools don’t go and ask for the data because the minute you do that you’ve blown it. Go and ask what evidence you’ve got and the same for professional development.What evidence can you produce before you do the PD and as you do it that you’re having an impact and what are you doing about the consequences of that. I’m suggesting that that will get away from the prescriptive notion, it opens up opportunities, it gives the communities that we work with who are incredibly diverse modern of ways of showing they’re successful but it does put the emphasis back on impact.
“Professional learning does work. But its least effect is on kids.”
So what does the PD research say? Well across the 50-odd thousand people that I’ve got in the sample that ask what the effect of the PD is, it’s pretty impressively good news. I don’t think we should ever imagine that professional learning doesn’t work, it does work, by my scale its way up there. But, like all things with but you never remember the word before it, I’ll remind you of the word before it – it’s variable. Its least effect is on kids.And I think that the mission we should be saying here in Australia is thanks to this [the Charter (PDF)], is how can we be having that kind of impact as well. And certainly this also says to me that we shouldn’t be measuring our impact solely in terms of teachers and any kind of evidence about the learning that accrued, how it changes their behaviour, how they felt about it. Not enough. Important but not enough but that 0.37 is below my line, not much, but it gives me hope that we can make a difference in that area.When I go further and ask what’s the most effective types it kind of looks, when you look down that list there, it kind of feels like teaching. It kind of feels like what good teaching is, because I’m going to contrast that with the least effective. And the other thing about this list is that there is a few of those that I hardly ever see.
“Poland and Finland have abolished any kind of notions of homogeneity: streamed schools, streamed kids.”
Now one of the things that actually enthralls me from what Barry McGaw is telling me and I see it in the evidence of PISA, I’m looking to Poland which is probably the fastest growing country in terms of their changes and I look to see, you ask the question what do they do that is different. It’s kind of what Finland did 30 years ago and probably some others, they’ve abolished any kind of notions of homogeneity. Streamed schools, streamed kids. I think that there’s a tremendous amount we can learn by mixing up kids and mixing up teachers. And I look at that number four there about mixing teachers across high and primary schools and you see that’s when you get major impacts, to me it’s that they’re bringing different world views together and they’re not assuming anymore.Let’s look at the least effective: Stuff. Resources. And so the message is that we shouldn’t stop for a moment when we look at professional learning and say it’s something we can do to people. It involves the same acts, quality of teaching that it does with kids. And I want to see more about this [in the Charter (PDF)]. It isn’t just about going out there and giving teachers’ stuff. It is about changing their minds, changing their views, gaining their minds, gaining their views to convince them it is about impact.I want to finish with my colleague, I’ve spent 12 years in the Auckland factory with people like Viviane Robinson, Helen Timperley, it’s great to go to work because like all good places they’re wonderful critics of your own work. And Helen has done some of the most impressive work and started off with the notion that I only want to ask the question: What professional learning makes the difference to kids learning?And as she started her work, and I was part of that team when it started, is that we ruled out anything to do with teachers. Not because they’re not interesting, the outcomes on teachers, but it wasn’t the focus. That got rid of most of the research. And she ended up with not as many studies, about 70-odd. And she again came up with similar conclusions that I showed there before in terms of the impact on teachers’ thinking, teachers’ knowledge is much higher than it is on the impact on students.And she did come up with these seven themes and I think they’re very powerful themes and one of the things I look for in here [the Charter (PDF)] is how is this mirrored. Now, it’s not here but I’m challenging you to take the next step and take this kind of work. How can we get the word out to our communities that there are some things that we know in this business. That we do know that just doing it in school may not have the sharp edge and it may not question the assumptions you’re making.
“Teachers talking to teachers about teaching.”
Now one of the pieces of work that I did many years ago with one of the Ministers for Education was closing schools, a word of advice, don’t ever do that. Even the worst schools, even when the community knew they were the worst schools, they weren’t going to change them. The fight to keep them was ginormous, beyond all comprehension. Even when some of them were toxic.And so when you go down to that school level in some of those systems and asking them what they want to do, you’re just inviting disaster.When you go into the schools that are the opposite, as I said before, you leave them alone and its almost that notion of earned autonomy that I tried to say here, when you look at notions like involvement of external experts. I don’t think it’s because the external experts are any good, I think it’s because you’re bringing in a different mindset.And that notion of teachers talking to teachers about teaching. The most anal study I’ve ever done, in fact it was with Helen, we measured in minutes what teachers talked about at morning tea time, lunch and PD sessions. In primary school it’s about three K’s: we love to talk about the kids, we love to talk about the curriculum and obviously in New Zealand we love to talk about kicking footballs. In high school we added assessment. One minute a month teachers talked about teaching. We almost have a whole psyche in our schools that says: ‘Michael, you teach different to me and I respect that.’ Now what I’m saying is: Don’t you dare touch me. And that’s one of our problems in our system is that that whole teaching act, not only is it private – we don’t even talk about it.
“How can we get away from ‘best practice’?”
Now, as an aside, we did that study because when we introduced our assessment package when the Minister asked me how would you judge you’re successful he expected me to say that I could meet all the Peter Hill standards of psychometric brilliance (which is my cup of tea). And I said to him, no that’s an assumed. My standard is: Do I increase the amount of time teachers talk about teaching?So we went in, and we did have success, so there is an explanation for anality but I still think its fascinating about how we can get that number five, how we can open it up. How we can get away from that crazy thing we were doing for 20 or 30 years about best practice. When teachers said, look, this is best practice and I’ll tell you about it and the whole message was that’s it. When they failed to realise that [the thinking/the mind] was it. It’s what they did to that practice and its how we can get to that teachers notion of talking about teaching is very powerful.
“How do you get away from ‘everything goes’ and the rhetoric of ‘what works?’”
So let me sort of end by giving you some challenges, AITSL. I personally think that this [the Charter (PDF)] is a major step forward. One of my challenges to you is that you’ve now done this, you’ve now done professional teaching standards, you’ve done it to leaders, I’m sure you’ve got some other things up your sleeve.My worry is that you’re creating the three legs of the stool and there’s no bench yet. So how do they relate to each other? How do they get one thing out of this so we’re not giving 14 messages to the school system? I think you’re right for that, I’m not asking you to do more work in one way, but I’d love you to think of that.I would also love you to think that notion forward is how do you start creating a debate around this country about effectiveness. How do you get away from ‘everything goes’, how do you get away from the whole rhetoric of ‘what works?’ It’s a disastrous language, because everything works. How do you start to say and help our communities identify what excellence means?
“Help schools evaluate the impact of their learning on the performance of the kids.”
Now my view is that what I’d love you to do is come up with ways to help schools evaluate the impact of their learning on the performance of the kids. I would hate for it to be prescribed, because there is clearly no one way and, furthermore, I know that it won’t be adopted. And so that notion of how you can start to build that evidence as we did at NCATE, through websites, as we did at NCA New Zealand, is to get that community out to say this is acceptable evidence that answers the first question: What is the impact of the PD on the students in the school? And then the second question again: How do you help get the consequences of that, where to next?Given we’ve gone and done this, how do you make it a systematic program and not a one-off program? And so, in my language, which is very narrow, is how do you get those effect sizes so that all our community, all our teachers join what I think is an incredibly successful business that we’re in and that we drive the system from the most successful and not put all the onus and see PL/PD as fixing the problems. I think this is a wonderful challenge – it is yours.”
About the presentation
John Hattie’s presentation was part of a one-day conference “Promoting a National Professional Learning System: a call to action” organized by AITSL which took place on 28 October, 2011 in Melbourne Australia. The draft transcript on which was used and edited (formatting, headings, hyperlinks, minor corrections) for this article can be downloaded as an RTF text document from AITSL’s website. The video of John Hattie and the other speakers were published on AITSL’s youtube channel.
The Australian Institute for Teaching and School Leadership (AITSL) is funded by the Australian Government. AITSL provides national leadership for the Commonwealth, state and territory governments in promoting excellence in the profession of teaching and school leadership.
AITSL works with the education community to:
- set and maintain standards to promote excellence in teaching and school leadership;
- lead and influence excellence in teaching and school leadership; and
- support and recognise excellence in teaching and school leadership.
For more information about AITSL visit www.aitsl.edu.au