Transcript of S2 E3: The Forgotten Experiment

Previously on Sold a Story.

The man came in with a cigarette hanging from his mouth.

What if I had something as hard as this?

5!

That's it!

Yo!

I couldn't believe what I was seeing, and I thought, this is the future. One educator has called the approach totalitarian, another horrible.

There's a wide range of people against it. It doesn't fit what they want education to be.

By 1967, President Lyndon Johnson was looking for more ways that schools could help children from poor families. His Head Start preschool program was already serving 3- and 4-year-olds, and he wanted to build on that with a huge new federal effort in elementary schools. But something happened. Something got in the way.

These ruins are in Saigon.

The Vietnam War.

And I said, where's all this money? We were promised an education. And he said, it's on boats going to Vietnam.

The president's education ambitions had to be scaled way back, and his plan for a big new program in elementary school got turned into something else: an experiment, a study.

It was like a Hail Mary. It was like a way to save the program.

The goal of the study was to answer that question on a lot of people's minds by the late 1960s: What actually works in education? How could school become a more powerful tool for helping kids get out of poverty? The study was going to try to find out.

And for the first time, research institutions, universities, others with theoretical and experimental ideas about education of children were asked to put them into practice in the public schools.

It didn't just evaluate different curricula or different instructional methods. It evaluated different philosophies of education.

The study became known as Project Follow-Through. The idea was to follow through on the promise of Head Start. The study began in 1968 and lasted nearly a decade. It was one of the largest educational experiments in the history of the world. Nearly 60 years later, that's still the case. And Project Follow-Through ended up providing some answers, some information that could have influenced the direction of American education. But it didn't. I'm Emily Hanford, and this is Season 2 of Sold a Story, a podcast from APM Reports. We're at episode 3. You've probably never heard of Project Follow-Through. Lots of people haven't, including teachers. Do you know about Project Follow-Through?

No.

Do you know about Project Follow-Through? I do not. Have you ever heard of Project Follow-Through?

No. No.

My producers and I spent 2 years figuring out what happened with Project Follow-Through. We tracked down researchers who worked on the study.

It is like a blast from the past.

You know, I haven't thought about Follow Through in decades. We went to the National Archives. I just found some documents from 1970. Where we dug through dozens of boxes of letters, memos, notes, reports.

The missing data that was never released to the public. Maybe.

Sure looks like it. In this episode, I'm going to tell you what we learned about Project Follow Through. And why some people didn't want it to influence the direction of American education. Government officials were scrambling to turn President Johnson's plan for a major new education initiative into a study. As they were searching around for which programs and educational philosophies to test, the head of the U.S. Office of Education went to a conference in New York. and heard Zig Engelman giving a talk about Direct Instruction. He went up to Zig after the talk and asked him on the spot if he wanted to be part of Project Follow-Through.

I confessed to him that I'd never heard of Follow-Through, and he confessed to me that he never heard of Direct Instruction.

Zig immediately called his co-director back in Illinois.

And I said, should we do it? And he says, yeah, let's do it.

But they had a problem. They didn't have a Direct Instruction program for elementary school. They would have to create one fast.

We were inventing the airplane while we were flying it.

You met Doug Carnine in the last episode. He worked with Zig.

It was a madhouse of writing the curriculum, figuring out the video production system to video teachers, send in the videos, send back feedback.

They were trying to keep up that routine from the preschool, where they'd teach, figure out the mistakes they made, and then revise the lessons. But now they had schools all over the country. There were Native American communities.

I went to New York. I went to Providence.

I went to Flippin, Arkansas, in the Ozarks.

Spanish-speaking sites in East Las Vegas, New Mexico.

Brooklyn.

Flint, Michigan.

Uvalde, Texas.

Dayton.

Smithville, Tennessee.

I flew all around the country for years.

Adrienne Allen is the woman who watched Zig walk into the classroom with a cigarette hanging from his mouth. She ended up working with him on Project Follow Through.

You know, you'd be at a site, like in New York, and then you'd get a call, we need somebody to go to Idaho, so get a plane ticket and go there.

You had to be ready for anything.

They would fly me down in a single-seater.

Jerry Silbert remembers flying to South Dakota.

And we land on some grass in a field, and the pilot said, oh, I hope there's no cows there.

Jerry was a graduate student when he started working on Project Follow-Through.

I was one of the older ones. I was 28.

I was 20, so—

Suzie Andrist was still in college.

I didn't come in with a lot of credibility.

It was a tense dynamic sometimes, these young people coming in telling veteran teachers to do things differently.

I had a couple of teachers who just were really negative.

I was hated a lot.

This is Adrianne Allen again.

Because I was a policeman, I went in and made sure that they were following the script.

There were teachers who did not like those Direct Instruction scripts. They wanted to do their own thing.

Some of them would lock the door and they would pull down the shade so that I couldn't see in the classroom or go into the classroom.

She had supportive principals who would help her out. But some principals were not supportive. And there were other obstacles. This is Gerry Silbert again.

I remember one site, we did a training for a week or 2. And then the teachers got transferred. So school starts and we're missing half the teachers who were trained.

Other programs being tested as part of Project Follow-Through were facing similar challenges.

I remember in one classroom—

Larry Schweinhart worked on a program called HighScope.

The teacher's doing business as usual. She looked up at me when I came in and she said, do you want me to do the HighScope now?

She wasn't doing the program. It's not clear why, but it's not that surprising that some teachers were reluctant. These were new, untested ways of teaching. It was up to the program developers to get teachers on board, to figure out how to make the educational ideas they believed in work in the real world. There were 22 different programs being put to the test as part of Project Follow-Through.

They represented just about every theoretical structure that existed for early childhood education.

Chuck Salzberg says the programs were all over the map. Some were highly structured, like Direct Instruction. Others were a lot looser. Some put a lot of emphasis on getting parents involved or motivating kids. Chuck Salzberg worked on one of those. The teachers are circulating, giving short instructions, doing what we call behavior-specific praise. Hey, Sue, you're reading that very well. You're doing better and better. The teachers would give out tokens that the kids traded for rewards, like a special toy at recess. And many of the programs were based on the popular idea you heard about in the last episode, the idea that kids learn best through experience and discovery. One of those was the HighScope program you heard Larry Schweinhart mention a moment ago.

It really focuses on children being empowered to make their own decisions and to develop their own plans and activities rather than the adults simply telling children what to do.

The various follow-through programs didn't just have different approaches to teaching and learning. They had different goals, different things they were trying to accomplish. Zig Engelmann's goal with Direct Instruction was to teach kids academic skills— language, reading, math. Other program developers had other goals. This is the creator of the HighScope program.

We want the child of independence, Of originality, of creativity.

Measuring all those things was going to be difficult. How do you know if a program has succeeded in helping a child become independent or creative? And even if you can reliably measure those things, how are you going to compare programs to each other if they have such different goals?

There was a huge amount of disagreement about What should be measured in the evaluation?

Bob St. Pierre worked on the government's evaluation of Project Follow-Through.

The issue of trying to get measures that treat everyone fairly, it's always an issue.

We could see the fights in the documents at the National Archives. The government initially focused its data collection on basic academic skills. That worried the developers of some of the other programs. They thought it would give Direct Instruction an unfair advantage.

It was easy to prove the effectiveness of Direct Instruction.

This is Larry Schweinhart again.

But it was difficult to prove the effectiveness in these other areas that we considered more important.

In response to complaints, the government added other measures. There were tests that looked beyond basic skills to things like abstract reasoning and problem-solving. And there were measures to assess things like self-confidence and attitudes about school and learning. Not all the program developers were happy with the measures that were chosen. They were still concerned that what they were trying to accomplish would not or could not be accurately measured. It was a high-stakes situation for them.

This is their life's work.

This is Bob St. Pierre again. He worked on the government evaluation.

You know, you've got these evaluators coming in and they're going to come up with a report that might say something bad about their life's work. Or something good.

The U.S. Office of Education was planning to distribute information about the program models that were most successful. This was an opportunity to have a huge impact on American education.

Everybody wanted to become the model that would become the dominant model.

Follow-Through was a horse race to compare these models to see what works.

But the idea that Follow-Through was a horse race that would have one winner That idea was going to be contested. And here's the thing. At the beginning of Project Follow-Through, many people, including government officials, were assuming there would be lots of winners.

The idea was that every model would be so successful it would just blow the doors off of the competition, the competition being Business as usual.

This is Cathy Watkins, who has written about the history of Project Follow-Through. She says the assumption that all the follow-through models were going to be better than what was already happening in schools— that assumption meant government officials were especially alarmed by what happened next. The first year of data arrived on their desks, and it did not look In fact, it looked so bad that they decided, we can't let people see this.

A decision was made that those data would never be released to the public.

We found that first-year data in the National Archives. It showed that students in the follow-through models were not doing better than students in traditional classrooms. Some of them were doing worse. That was bad news for the government. Because the goal was to find better approaches. But Zig Engelman wasn't bothered by the early results. He thought it was going to take time to get Direct Instruction going, and he was feeling confident. So were his colleagues.

The program worked well after several years, and it worked well because the teachers got good. They were saying, you know, this program teaches kids I've never been able to teach before.

They'd be in the lowest group, And then they'd move up and up and up.

And Zig was telling them, your hard work will pay off.

They said, it's going to be worth it. You know, it was hard, but they said, we'll be able to help lots of kids.

It was pretty exciting to think about how this might grow and how many more kids might get taught.

The government collected data for nearly 10 years. It had test scores and other information about more than 350,000 students in thousands of classrooms all over the country. It released some of that data to the public and the program developers as the study was going on, but the ultimate verdict on the effectiveness of the different follow-through models was going to come down to a final evaluation. Coming up, The government releases the evaluation, and there's a winner. But that's not the end of the story. That's after a break. By 1977, the Project Follow-Through study was wrapping up after nearly a decade, and the government was getting ready to release its final evaluation. A Direct Instruction trainer named Marcy Stein was on her way home from a school in Flint, Michigan, where she'd been coaching teachers. At the airport, she ran into one of her colleagues. And he said, oh, I just came from the meeting. The meeting. The meeting was for the developers of all the follow-through models to give them a preview of the final evaluation.

I said, well, what happened?

And he said, well, you know, we did really well. He told her the other program developers were shocked. He said the place went nuts.

They couldn't believe—

that's, that can't, that's gotta be wrong.

We beat 'em in everything.

The government's evaluation showed that Direct Instruction had outperformed all the other models. It had done especially well compared to the other models when it came to math, reading, and spelling skills, exactly what Direct Instruction was trying to do. But it had also done well when it came to things like abstract reasoning and problem solving. And perhaps the biggest surprise? Direct Instruction performed the best on those measures of self-confidence and how kids felt about school and learning. Even Zig was surprised.

The DI model never professed to be concerned with self-image, and yet our kids had the best self-image. Well, guess why? They knew damn well they were smart.

Zig was triumphant.

I remember that everything was electric.

Zig's son Owen was a teenager at the time.

This is awesome. It's going to be disseminated everywhere.

The assumption was that once the data got out, then the Direct Instruction model would be used widely all over the country with disadvantaged children.

This is Doug Carneyn again, Zig's colleague.

I mean, that was the whole purpose of doing it. It wasn't some kind of academic exercise.

But there were 2 groups of people that did not like what the final follow-through evaluation said. The first group that didn't like it? The government itself, the U.S. Office of Education.

It didn't say what any of us were hoping it would say.

Remember, they were hoping all the models were going to be better than what was already happening in schools. But Bob St. Pierre, who was one of the authors of the government evaluation, says that's not what the data showed. The data showed that Direct Instruction was the only model that had positive results in every category that was measured. In all the other models, the kids did worse on average on at least one set of measures. And that brings us to the other group of people who were unhappy with the results: the program developers who did not do well. They complained to the Ford Foundation. It had been funding some of the unsuccessful models. And a program officer from the foundation made a phone call to a guy named Ernest House.

So she called me and says, you know, I'm getting a lot of complaints from a lot of the follow-through sponsors saying they're being unfairly treated.

The program developers were also known as sponsors.

I don't know what to make of this, she said. So what do you think?

Ernest House was one of the country's leading experts in evaluating social and educational programs. He agreed to meet with the sponsors.

And I went out to this kind of rural retreat they were at.

It was a summer camp in Michigan run by the man who created the High Scope program. It was a rustic spot— cabins, bunk beds, a dining hall.

I asked them what they thought, sat down with them over coffee. I listened to their meetings.

Some of the program developers were still upset about the measures the government had used, felt that what they were trying to accomplish with their models hadn't been accurately measured. And there were some issues with the study design that went back to the fact that Follow-Through was originally supposed to be a big government program, not a research project.

So after listening to them, I thought, I think they got a point here. I think they got a point.

Ernest House convened a group of experts to review the government evaluation.

We went through the whole thing.

They thought it was a messy study that could have been set up better. The measures weren't perfect. And they concluded some of the government's statistical analysis was flawed. So House and his group made some adjustments and ran the numbers again. Some of the models did better, and some of the models did worse, but the winner didn't change. Direct instruction still came out on top. However, Ernest House and his panel came to a very different conclusion about what it all meant. They wrote a critique of the government's report and took issue with how the government had defined success. This is a little complicated. Bear with me. The government evaluators had calculated the average performance of the various educational models. They took all the data from all the places where each model was used and figured out how effective each model was on average. It's kind of like rating restaurant chains by averaging customer reviews from every location across the country. Overall, some chains are better than others. But that doesn't mean every location of a highly rated chain is good, or that every location of a poorly rated chain is terrible. Same thing happened in Project Follow-Through.

According to the government's evaluation, the models that didn't succeed, on average, had some schools that did relatively well. The teachers, the coaches, the principals, They made it work. And the models that did succeed, on average, didn't succeed everywhere. Even Direct Instruction had schools where things did not go well, where kids did not do better. And that's what Ernest House and his group focused on in their critique of the government's evaluation. They focused on the variation.

You take one program, put it on the east side of Manhattan, you get one result.

This is Ernest House.

You put it in Urbana, another result. You put it in East St. Louis, yet another result. The very same program. That's the lesson from Follow Through. There's all this variation. Therefore, you cannot draw the conclusion which is best.

Project Follow Through had not revealed a program that worked equally well everywhere, every time it was implemented. And for that reason, Ernest House and his group concluded that there was no Project Follow-Through winner. There couldn't be a victor.

I remember Ziggy's rage.

That was one bitter-ass pill.

Zig and his colleagues fought back. They contacted politicians, wrote letters. Eventually, the U.S. Commissioner of Education responded. He acknowledged in a letter that Direct Instruction was the only program that consistently produced positive outcomes. But he wrote that because Follow-Through had identified only one successful program, the U.S. Office of Education was forced— that's a quote— forced to abandon its plan to distribute information about any of the programs.

We have this in a letter that he wrote. We can't call anybody a winner because there was only one winner. I mean, come on, man.

The government's logic was that, as the federal government, it could not promote just one educational program. Local control is a foundational principle of American education. Decisions about what and how to teach are supposed to be made locally, not by the federal government. But Zig and his colleagues thought there was more to it than that.

We were the opposite of what the establishment thought education should be.

Gary Davis worked with Zig. Like, it should not be structured.

You shouldn't be teaching kids. The kid will learn in their own time.

Their logic goes something like this.

This is Zig again.

Direct instruction is at odds with my model of the mind and learning, which is built My model is built around discovery. If I accept data showing that DI works, I will have to acknowledge that discovery and my model of learning is flawed.

There's no question there were influential people who didn't want the nation's schools using Direct Instruction. One of them was a man named Gene Glass. And Dr. Glass, are you okay with me recording this and using it in the podcast?

Yeah.

This is our producer, Emily Havoc, interviewing Gene Glass. Can you just introduce yourself, say who you are and kind of what you do or did?

Gene Glass. I've worked on problems in education for, Lord, since the early 1960s.

Gene Glass is a towering figure in the field of education research. He pioneered new methods for evaluating programs. But before that, back in the 1960s, he was a young professor at the University of Illinois, at the same time that Zig Engelman was starting his preschool there. And Gene Glass went to see Zig's preschool. He told us he watched a lesson for about 10 minutes, and he didn't like it. It seemed to him like the kids were being tormented. That was his word.

I think Performance and achievement and outcomes depend so much on things other than shut up, sit still.

This was his impression of Direct Instruction when, more than a decade after his visit to Zig's preschool, Ernest House got in touch with him. House invited Gene Glass to be one of the experts to review the government evaluation of Project Follow-Through. Gene Glass had nothing good to to say about Direct Instruction when we interviewed him.

I grant them nothing.

Nothing.

I find no evidence in the entire follow-through evaluation that they have anything special or worthy of wide distribution. It's a terrible way to organize teaching and learning. The best education starts with the interests, the motivations, the curiosity of the Sit down with the child, find out what the child is interested in. Let the child bring his own book, the book he loves, you know, see if he can read, see if he understands.

After the critique by Jean Glass and Ernest House, the government did 2 things. Number one, it started giving more money to the follow-through models that had not done well. The rationale was that those models needed more support, more time to prove themselves. And number 2, the Office of Education shifted its focus from highlighting successful follow-through models to highlighting successful follow-through schools. By doing this, it was possible for the government to showcase many of the follow-through models because most of them had a school somewhere that was doing reasonably well.

Ziggy was always going around saying, oh, you guys heard us, you did us in, we won the horse race, and so on.

This is Ernest House again. He acknowledges that the follow-through critique he wrote with Gene Glass had a huge impact.

I do know our critique was very effective in stopping the government from saying direct instruction, that's the way to go.

We asked Bob St. Pierre about this, the government evaluator. He agreed the government's response to Project Follow-Through should not have been, Direct Instruction is the answer. Schools should drop what they're doing and adopt Zig's program. But he thinks it was a mistake for federal officials to basically ignore the fact that Direct Instruction did better, on average, than all the other models. He says averages matter. They tell you something.

If you don't believe in looking at average effects, What's your conclusion if that's what you think, that, that we can never learn anything?

The message after Project Follow-Through was anything can work somewhere. And that gave people permission to pursue what they liked, what they believed in, what they thought was best for kids. But Project Follow-Through had provided some clues about what works in education. The model that did the best, after nearly a decade with lots of kids in lots of schools in one of the largest education studies of all time, was a highly structured program that relied on carefully sequenced lessons and didn't expect students to discover things on their own. It was an answer, but it was not an answer a lot of people wanted to hear. The end of Project Follow-Through was devastating for the people involved with Direct Instruction.

Probably the heartbreak of my career.

Susie Andrist was that 20-year-old college student. She decided she was going to leave education altogether.

I was just sick of the whole scene.

She went to school to study computer programming. She says it seemed like a more rational profession.

In education, it was just like people just did what they wanted to do, you know, regardless of the data.

But Zig wasn't giving up. His son Owen says what happened after Project Follow-Through made his dad even more determined.

He was like going, we're gonna show those MFs. We're gonna show 'em. We're gonna do it. He's all in. He's all in.

Zig Engelman didn't give up. He kept teaching kids and writing programs for the next 40 years. And his ideas spread. On the next episode, we take you to an elementary school in Texas that started using Direct Instruction back in the 1970s. Almost all the kids were Black and from low-income families. They eventually did so well, people thought they must be cheating. We've had them come in and say, we think they're cheating at Wesley.

Can you find out how they're doing it?

That's next time on episode 4 of Sold a Story. When we were digging through all that material about Project Follow-Through in the archives, one thing we could find almost nothing about was what it was like for the students. So we tracked down a bunch of former students and asked them. You can read what they had to say on our website, soldastory.org. Sold a Story is a podcast from APM Reports. It's reported and produced by me, Emily Hanford, with Emily Havoc, Olivia Chilcote, Carmela Wallianone, Kate Martin, and Christopher Peak. Our story editor is Curtis Gilbert. We had additional editing from Sasha Ezlanian. Andy Cruz is our digital editor. Mixing, music, and sound design by Chris Julin. Fact-checking by Betsy Towner Levine. Our theme music is by the band Wonderly. Final mastering of this episode by Mauricio Dorico. Our executive editor is Jane Helmke. Special thanks to Emily Corwin and the APM Reports team. Thanks also to Kate Kelly, Dan Crocker and Margaret Goldberg. Leadership support for Sold a Story comes from Hollyhock Foundation and Oak Foundation. Support also comes from Ibis Group, Esther A. and Joseph Klingenstein Fund, Kenneth Raynan Foundation, Wendy and Stephen Gahl, and the listeners of American Public Media.