Wednesday, October 19, 2005

NAEP scores just out; Oregon about average

The NAEP scores were released today. For the uninitiated, NAEP stands for National Assessment of Educational Progress. They are snapshots of average achievement levels in each state. Every student doesn't take the NAEP - it is only given to samples of students, and only to 4th, 8th and 12th graders every few years. The results are, however, statistically significant indicators of a state's achievement, and can be compared state to state.

Of course comparing states is still tricky, because of differing demographics. Affluent states with fewer minorities will tend to score higher than poor states with large minority populations. Oregon has an advantage on the demographic scale. We have a mostly white population, relatively high income, and higher than average education level.

But despite our demographic advantages, we score just about average on the NAEP. In some areas, slightly above average.

Scores were released for 4th grade reading, 8th grade reading, 4th grade math and 8th grade math . In general, all the scores were slightly down from the last time the NAEP came out, 2003. Compared to the first years Oregon participated, in the 1990s, most of the scores are up.

The exception is 8th grade reading, which for some reason has pretty much steadily declined since 1998.

I've already seen some press releases on the scores touting the fact that Oregon's scores are higher than the mid-1990's scores. That is all well and good, but it glosses over the troubling fact that there is an apparent contradiction between NAEP and Oregon's statewide test results over the last couple years on the math test. That is, the math scores on Oregon's test have gone up significantly, while the scores on NAEP have not.

NAEP tests in grades 4 and 8, while Oregon tests in grades 3, 5 and 8. The results are expressed in terms of the percentage of students who are "below basic," "Basic," "Proficient," and "Advanced" in achievement. Oregon's tests have three categories - "Does not Meet," "Meets," and "Exceeds" benchmark. To compare the two tests, let's look at each test's results for the percentage of students who are at or above the "Basic," and "Meets Standard" levels.

Between 2003 and 2005, 8th graders on the Oregon's math test did better by 2.5%, but on NAEP, they did worse by 2.5%. Fourth graders on the NAEP math test did better by 1%, but on Oregon's test the third graders grew by a whopping 7.2% and fifth graders by 5.0%.

What is going on? I've argued for years that Oregon's assessment system is not reliable and valid. In 2003 we released a study by an OSU economist that had compelling evidence that Oregon's math tests in particular had gotten easier over the years. This gives us one more piece of evidence that it has.

The Oregon Department of Education has consistently ignored the mounting evidence that something is wrong with its tests. The mainstream media rarely asks penetrating questions about their claims of validity. How long can there be such a stark divergence between Oregon's scores and the NAEP scores before they have to actually admit that something just might be wrong?

Don't underestimate the ability of bureaucrats to stonewall and obfuscate. In Oregon, it is an art form.

6 comments:

Bailie said...

Thank you Rob,
The results are interesting. So tell me again why Oregon's K-12 employees are the 8th highest individually compensated of all states? At the expense of having more teachers (K-3), complete school years and programs.

Rob Kremer said...

That is the easiest question I've been asked all day. The answer is simple: The OEA is a very effective traditional labor union that happily trades higher class sizes for increased pay and benefits.

Anonymous said...

"""The OEA is a very effective traditional labor union that happily trades higher class sizes for increased pay and benefits."""

That's not very nice.

gus miller said...

Rob:
You make some sound observations about the divergence of Oregon's test results from the NAEP test results. There is also the PALT testing that Portland District 1J administers to its students.

I firmly believe that Oregon classrooms would benefit from cost savings that would accrue from purchasing and utilizing nationally normed off the shelf tests.

gus miller said...

Anonymous said it is not nice to say: """The OEA is a very effective traditional labor union that happily trades higher class sizes for increased pay and benefits."""

Reality is not always nice. Nor are OEA demands to restrict the hiring of new teachers or transfers of veteran teachers in many districts. Nore are the
teacher pay scales based upon seniority, degree status and additional credits with elementary school teachers compensated at the same level as Advanced Placement Math and Science teachers.

Tony said...

Rob,

Maybe they need a challenge to administer the tests they gave in, say, 1980, or even 1990. That would be an interesting result, I am sure.

That would be great in a run for Superintendent, now wouldn't it? Now if only we had a canadidate...know anyone?