CIM/CAM and HB 3162

If you really want to know about CIM/CAM, read this. It is long, but that is because there is a lot to explain. I’ve worked on this issue for more than 5 years, and there is very little reason for anybody who is not in some way directly involved in the policy-making end of education to know much about the complexities of all the aspects of CIM/CAM, educational assessment, standards, and the like.

For those who want to know all the nuances, I offer the following. I do not try to hide my opinions that are liberally interspersed through the piece, and I stand behind every single factual statement that I make.

I wrote HB 3162, which would eliminate CIM and CAM, and replace Oregon’s troubled assessment system with a valid system designed by an independent testing company. Of all the political efforts I have been involved in, the effort to end CIM/CAM has to be the most frustrating.

The battle lines are very, very interesting. It is completely non partisan issue everywhere except the legislature. Parents, teachers, students who have direct experience with this failed reform are overwhelmingly in favor of getting rid of it.

Those opposed to getting rid of it are almost exclusively those who have been involved one way or another in implementing it. This includes ODE bureaucrats, district administrative bureaucrats, and the pretend business organizations such as Oregon Business Council, Oregon Business Association, Associated Oregon Industries, etc. Also supporting the status quo are all the pressure groups who feed off the education system – OSBA, COSA, OEA.

So basically we have an entrenched system that has been re-designing the methods and purposes of our school system for more than a decade. They have lots of bureaucratic inertia, lots of vested careers, and a track record with identifiable failure after identifiable failure.

Uprooting this disaster is not easy. In fact, I am not at all confident that it will be done. Much more likely: the CIM/CAM grip on the schools will get firmer and more intrusive, increasing further the public disenchantment with a disconnected school bureaucracy that refuses to listen to its customers.

I'm going explain a few things about CIM/CAM and what HB 3162 proposes. If you already know all this stuff, just skip it.

A Brief History of CIM
CIM/CAM came to Oregon with Vera Katz’s “21st Century School Act” in 1991. The Act front-ran the Bush 41 “America 2000” initiative and Clinton’s “Goals 2000” program, both of which basically gave incentives for state governments to establish “standards” for the schools and then design a system of tests intended to see if students met the standards.

OK so far. I’ve got no political or theoretical problem with this concept, although there is an argument to be made about local control, school district vs. state authority, etc. But let’s leave that aside for now.

So, Oregon was one of the first states to adopt a program of state-defined standards and tests. Why were we first? Interestingly, it was because the primary policy work surrounding the America 2000 and Goals 2000 programs were done on the east coast by a think tank called “National Center on Education and the Economy” (NCEE) upon whose board sat Hillary Clinton, Ira Magaziner, several corporate titans (including John Sculley, former Pepsico chief and Apple Computer CEO) and the Oregon connection, Vera Katz.

In the late 1980’s NCEE printed a white paper called “America’s Choice: Low Skills or High Wages,” which proposed a redesign of U.S Public schools around a skills/job training model. Schools would teach and certify students’ acquisition of the skills necessary for the job market.

Guess what phrase NCEE coined as the very first certification students would receive from their job training in school? Yup, you guessed it: Certificate of Initial Mastery.

Vera brought this conceptual design to Oregon in 1991, and got it through the legislature through a backroom deal with Larry Campbell, who had just deposed Vera as speaker. There was a good bit of resistance at the time. The House Education Chair, Rep. Oakley, refused to hear the bill at first, until Campbell strong-armed her and threatened to take her chairmanship away unless she went along. (She has confirmed this to me.)

So it passed, and the bureaucracy started churning away. There was lots to do. First the state had to develop all the “content standards,” and then devise tests to give students to see if they have met them. This is where we started going astray.

Big questions arise at this point – questions with very serious implications. The very notion of “content standards” implicitly involves defining what it is kids should know and be able to do at each level of school. This is where it gets political.

Whose views about what it is important for kids to know should become part of the standards? It’s a big problem.

After Goals 2000 passed in 1994, the Feds started funding all sorts of projects to create “National Standards” and gave money to states to help them in define standards. One condition of course was that the state standards had to be consistent with the national standards.

So now we’ve moved the very important question of what kids should be taught not just from the district to the state, but now to the feds – and really to various unaccountable professional organizations such as the National Council of Teachers of Mathematics, and their equivalent organizations in science, English, history, etc.

You may not know it, but these organizations are overtly ideological. They take positions on cultural and political issues all the time, and their views got reflected in the standards they wrote. Even such seemingly non-political issues like how to teach reading and math are in actuality intensely politicized and controversial.

Virtual wars have raged for years over phonics vs. whole word approach to reading, and traditional vs. fuzzy math.

Well, the professional organizations developed their standards, and they were uniformly on the “progressive” end of the spectrum on all of these issues. For instance, the national math standards recommended that "Calculators must be accepted at the K-4 level" and that any "further efforts at mastering computation skills are counterproductive."

The national history standards were so offensive in their anti-American politically correct slant that the US Senate denounced them in a 95-1 vote. The single no vote was cast by a senator who believed that simply denouncing the standards was not nearly a strong enough statement.

So anyway, Oregon went down this path a little ahead of the rest of the nation, and was the one state that used the NCEE model as its structure. (At the time they all said we were the “national model.” Ira Magaziner visited Oregon and touted our leadership. Like so many other Oregon “models,” nobody has bothered to copy it. Can you be a model if there are no copies?)

By the late 1990’s Oregon had all its content standards in place.

Oregon’s standards aren’t very good. Various rating agencies have pretty much panned them. But they are in place, and they define literally hundreds of things kids should know in every content area at every grade level. They have, however, gotten somewhat more rigorous over the years as they have been revised.

Assessment Oregon Style
At the same time, the ODE designed its tests. This is where it made a major mistake. Testing, like everything else in education, is highly political. Some educators believe that tests are useless, and there is now way any test can reveal anything of importance. Others believe that they can design valid tests to measure things such as “critical thinking,” “problem solving,” “higher-order thinking skills,” etc.

My opinion is that tests are very useful at measuring acquisition of essential academic skills and abilities, but should be used only to measure those things we know for sure we can measure with reliability and accuracy.

Oregon’s assessment bureaucracy has a view on this issue, and that view has become codified in the system of tests they developed. They designed a combination of 1) multiple choice tests that supposedly measure basic skills in reading and math, and 2) types of tests that are known as “performance assessments,” which they claim measures abilities such as problem solving and writing.

There have been constant problems with both Oregon’s multiple choice tests and the performance tests.

The performance tests are especially problematic. They claim that such tests, such as the Math Problem Soving (MPS) tests, are a good measure of deeper mathematical reasoning, but there is absolutely no evidence to back this up.

The MPS tests involve a multi-step story problem of which the point is NOT that the student must get the right answer. Click here for an example of a 4th grade test question a student has actually answered with all the pictures and diagrams. You really have to see it to believe it. Remember - this is a FOURTH GRADE test question, proudly presented by the ODE bureaucrats as a shining example of the knowledge they believe 4th graders should know.

The actual math involved in the problem is trivial. What deeper level of understanding do they imagine such an inane problem is revealing?

In the MPS test the student is supposed to analyze the math problem and then make graphs or charts or illustrations to prove he understands the concepts, and then write a narrative explaining how he attacked the problem. The teacher scores this part by rating on a 1-6 scale the kid’s “conceptual understanding,” and “processes and strategies.” The teacher then scores the narrative on a 1-6 scale according to how well the student “communicates” the answer.

Finally, the student is supposed to “verify” the answer through some other method, and the teacher scores the verification, also on a 1-6 scale.

Does this sound like a valid test to you? There are so many problems with it that I’m not sure it is worth going into them all, but if you’ve read this far you are a glutton, so I’ll explain some of the problems.

1) We are not measuring math ability here, but rather we are measuring communication skills and calling it a math test. An economist from OSU did a correlation of scores on the math problem solving (MPS) test to scores on other standardized tests of reading and math. He found that the MPS scores were far more correlated to language ability than to math ability. Why would we give a language test and call it a math test?

2) This type of test can only measure a miniscule amount of math knowledge. This makes sense, since there is only a single question asked. (Actually, the students get to choose one of three different problems to solve.)

3) The subjective scoring absolutely guarantees that there will be inconsistency in how the tests get graded. The ODE has spent huge sums of money training teachers in how to make very subjective judgments. For instance, the scoring guide delineates whether the “pictures, models, diagrams, and/or symbols used to solve the task are complete” (which scores a 4 on the “processes and strategies” category) or whether they are “fully developed” (which scores a 5.) I’ve talked to the assessment people at ODE about this and they claim up and down that this is valid and reliable. They are wrong.

4) The cost of putting this all in place is far, far greater than the cost of administering multiple choice or constructed response (where students fill in blanks with words) tests. All the teachers have to be trained and re-trained in how to teach kids what to do, how to score the tests, etc. Then the cost of the scoring itself is monumental. The ODE spends about $7 million each biennium on performance assessment related activities.

5) Oregon has had constant validity and reliability problems with these tests, especially the MPS test. In 1998 they had to adjust the scores of the 8th grade MPS test because they made the question too tough. Last year they had to throw out the entire 10th grade MPS test, and then suspend the test for all grades, because only 17% of 10th graders met “standard.” This cost millions of dollars, yet not one assessment expert at ODE lost his job. Now they are trying to figure out how to re-design the tests so that they are reliable and valid. Earth to bureaucrat: IT CAN’T BE DONE!

If all the above wasn’t enough, here’s the REAL problem with these assessments. Take a fifth grade math whiz – she’s top of her class, brightest, way ahead of her peers in what she can do. Give her the math problem solving test. She’ll fail it. What kid would know what the bureaucrats want her to do without having extensive instruction in the process of drawing pictures, diagrams, tables, writing the narrative, verifying the answer, etc?

So, the only way teachers can have a glimmer of hope that their kids will do well on this kind of test is to take class period after class period to train them what they scorers will be looking for. And this is just what they do. In fact, the ODE itself recommends that math classes take one day a week to practice the format of the math problem solving test.

So, we are basically forcing our teachers to take vast amounts of time away from instruction on important math concepts and use it to teach kids how to go through an entirely artificial and faddish process for solving a math problem.

This is a cost that can never be recovered. A day out of the instructional life of a grade school student can not be replaced. They only get one time through the system, and we cannot afford to waste day after day teaching them stupid processes rather than essential knowledge and skills.

Yet that is what our system of tests does, and that is the real problem with it.

So, the “performance assessments” have to go. They don’t work. All the ODE assessment experts will object, they will whine, they will say that we cannot back away from our commitment to measuring deeper conceptual understanding, but they are wrong. Remember – these are the same folks who designed the tests that they themselves had to throw out because they weren't valid and reliable. Why would we trust them?

Now, onto the multiple choice tests, where Oregon again made a huge mistake.

There are many companies who have spent decades developing tests that measure English and math skills. They’ve spent millions writing and testing the questions, figuring out exactly what content the questions measure, calibrating their difficulty, testing their reliability and validity, and doing all the other wizardry known as psychometrics. (For you real wonks – reliability and validity have scientific definitions. Plainly put, reliability means that if the test gets taken over and over by the same person it will get basically the same result. Validity means it actually measures what it purports to measure. So, for instance, Oregon’s math problem solving test has low validity because it measures verbal ability more than math ability.)

These companies have huge banks of questions, all of which have gone through their extensive analysis, and from which they draw when printing a version of the test. Academic assessment is a high fixed-cost/low variable cost business. That is, there is a huge up-front cost to develop all the questions and assemble the individual questions into valid, reliable tests, but the incremental cost of giving one more test is relatively tiny.

So, what did Oregon bureaucrats do when they were designing Oregon's testing system? Did they take advantage of the huge fixed costs already incurred by the testing companies, and contract with them to use their question bank to design a test targeted to Oregon’s content standards? No, of course not.

How many bureaucrat jobs would that create? (Oregon right now has 22.5 FTE in the ODE assessment division.) The ODE never met a wheel it didn't want to re-create. After all, Oregon bureaucrats can do the extremely technical and expensive job of creating valid, reliable tests so much better than the testing companies that have collectively spent billions of dollars and hundreds of years doing it, right?

So, we went trippingly down the path redesigning the wheel, and when all was said and done, we got a beautifully designed, very very expensive, square wheel.

Oregon’s multiple choice tests do not allow us to measure how Oregon’s students do compared to kids in other states. If you were going to design a test to measure 3rd grade math ability, wouldn’t you be interested in how Oregon compares to other states? Apparently ODE is not.

How do we know that the difficulty of Oregon’s math tests or reading tests have remained the same year in, year out? Because they say so? If the percentage of Oregon 3rd graders meeting reading standard has gone from the high fifties to the low eighties, as they claim, how do we know that isn’t because the test got easier?

We don’t. We have to trust them. I don’t know about you, but I do not trust state department of education bureaucrats to do anything other than to perpetuate their jobs. Remember – these are the very same people who designed the math problem solving tests that had to be canned, and then they recommended they spend a couple million dollars to get a do-over.

We actually have pretty compelling evidence that these “experts” have manipulated the difficulty of Oregon’s tests over the years in order to create the illusion of test score gains

If you look at Oregon math and reading test scores statewide from the mid 1990’s through 2002 or so, you will see a steady but slow increase until late 1990’s, then basically a spike, and after the spike, the steady slow increase again.

The OSU economist I mentioned thought this looked suspicious. That is not how large scale test data would normally behave. So he did another interesting study. He compared student results on Oregon’s multiple choice tests over several years to those same students’ scores on another test – the Terra Nova – which they also took those years.

You would expect the scores on the two tests to track pretty much the same, because the two tests are highly correlated. But he found basically that the same spike in Oregon-designed test scores in the late 90’s but the same students’ scores on the Terra Nova had no such spike. So it looks as if for one reason or another, Oregon’s tests got easier in the late 1990’s which explains at least part of what the ODE touts as increases in student performance. You can read the study yourself at OregonEducation.org

We pointed this out to the ODE at the time. They ignored it. We asked them for copies of prior years’ tests so we could do an item analysis that would reveal any difference in difficulty, and they refused. We asked them to prove us wrong by giving two different years’ tests to a group of students, and they refused. We got the old stonewall. Were they hiding something?

As a matter of public policy it makes no sense at all to have the state design tests that measure the success of the state’s school reform initiative. Do we let agencies audit themselves too?

So, the situation we find ourselves in, 14 years after Vera ramrodded the reform through the legislature: we have mediocre content standards, and we are proud owners of a slipshod basket of multiple choice and performance assessments that are very expensive, that we cannot trust, that give us very little useful data, and that consume huge amounts of classroom time that should be devoted to instruction.

So, that is the system of assessments that we have now, that HB 3162 would replace.

The CIM Design

And that brings me to the CIM. What is the CIM, anyway?

The CIM is basically the 10th grade assessment. The assessments at grades 3, 5 and 8 are as I described – a math and English multiple choice test, coupled with a math problem solving test in the 8th grade (it used to be given in the 5th grade too, and it is currently suspended,) and a writing test in the 5th, and 8th grade.

But in the 10th grade they really went full Monty. They designed the educrat’s dream assessment system for our 10th graders, a system that allows them to try out all their fanciful theories about how to measure all the meaningful things they want to measure.

As it now stands, the CIM involves multiple choice tests in reading, math and science, and then the fun begins. To get a CIM, students must complete their “CIM portfolio.” The CIM portfolio includes three classroom “writing samples,” three classroom “speaking samples,” and two classroom “mathematics problem-solving samples,” and one “science inquiry sample.”

The teachers job is to oversee all this work and then score it according to the scoring guides the state has developed.

So, lets run the numbers: take a 1000 student high school with 250 sophomores. All told, that makes 2250 CIM items that must be collected and scored if all the 10th graders are to get their CIM. But the scoring is the easy part.

If all the sophomores must give three speeches - that’s 750 speeches, given in class while the other kids watch. Assume you can get through 6 speeches in a class period. That means 125 entire class periods are devoted in a given high school’s sophomore class to watching each other give speeches!

That give you some idea as to the impact this has on what high school students spend their days doing. Talk to teachers – they know this is a waste of time. Talk to students. The problem is that the system the bureaucrats have devised is unworkable everywhere except in the minds of the bureaucrats.

And again – the true cost is not the money, but it is the time wasted that could have been used for productive learning activities. That cost can never be recovered. It dwarfs the actual direct dollars that are spent on CIM, although the direct dollars are considerable.

High school students know this is such a waste of time that only 30% of them bother to go through all the hassle to get the CIM. And that is for graduating seniors – this is a 10th grade assessment, but fewer than one in three get their CIM by the 12 grade – and that doesn’t take into account the 25% or so who have dropped out by then.

The CAM – The Next Boondoggle
Faced with the monumental failure and waste of opportunity of the CIM, the state is forging ahead with the next big idea - the CAM, or the Certificate of Advanced Mastery.

The CAM basically takes all the flaws of the CIM and takes it to the 10th power. I am constantly amused when I talk to people who are supporting the continuation of this stuff, people like Associated Oregon Industries. They say they support the CAM. I always ask: have you read the current “CAM Design Guidelines" that the State Board of Education adopted a few months ago? I have yet to have a single person know what I am talking about. Yet they support it. Totally irresponsible.

To believe what they have planned for our high schools with the CAM, you simply have to read this document. Words simply cannot describe it. The people who dreamed it up are living in some kind of Alice in Wonderland dream state.

The CAM starts when a kid steps into high school. At that point, he’s supposed to sit down with a counselor who has been trained in the newly developed “Comprehensive Guidance and Counseling Framework” to develop the student’s “Education Profile,” and his “Education Plan.”

The Plan is supposed to be a comprehensive roadmap for his high school years, taking into account his “strengths, likes and dislikes, aptitude, talents, learning styles, and skills through self-awareness activities, ” and laying out his short and long term “personal, educational and career goals.”

Stop right there. They have already proved that they are not to be taken seriously. There really is no reason to go further except for the sheer entertainment value of it. Remember, we are talking about 14 year old kids here. I have a 14 year old boy. He’s gunna be a major league baseball player. ‘Nuff said.

But wait there’s more. After our 14 year olds map out their journey through the wonders of CIM and CAM, they are directed to document their “personal progress and achievement toward: CIM academic standards, Career-related learning standards (CRLS) , Extended Application (EA) standards, and Career-Related Learning Experiences (CRLE.)"

What is all this stuff, you ask? That’s what the CAM is all about. It establishes a whole new basket of “standards,” all of the subjectively measured by the same type of scoring guides we have in the CIM, and almost all of them have nothing whatever to do with academics.

As you read this, please remember I am not making this up. I’m taking it directly from the CAM design document that you can get right off the ODE web site.

OK, here goes - Career-related learning standards: There are six so-called “standards” that students must meet in order to be certified that they are “career ready.” Remember: each of these so-called “standards” will have a basket of assessments, all of them subjective.

The standards are 1) Personal Management; 2) Problem Solving; 3) Communication, 4) Teamwork; 5) Employment Foundations, and 6) Career Development. Each of these has a definition, and then a set of “criteria” that schools are supposed to use when developing the assessment to see if kids have met the standard.

I love some of the “criteria.”

Under “Personal Management” is a criteria: “Take responsibility for decisions and actions and anticipate consequences of decisions and actions.” Oh I’d like to see a valid assessment for THAT.

Under “Problems Solving” is the criteria: “Identify problems and locate information that may lead to solutions.” Huh?

Under “Teamwork”: “Identify different types of teams and roles within each type of team;
describe why each role is important to effective teamwork.” OK, Johnny, now what type of team is this? Basketball! Good Johnny! Now, why is the forward’s role important?

Under “Employment Foundations”: “Identify parts of organizations and systems and how they fit together.” What can they possibly be talking about?

So, as the poor student is trying to navigate all these vague and meaningless career standards, he is also supposed to be pursuing his “Extended Application Standards.” These are standards that show the student has taken the skills he learned in the CIM (such as sitting through endless speeches of his peers, a skill that state department of education bureaucrats obviously have in great quantity,) and can apply them on an “extended” basis.

This is supposed to be proven through “collections of evidence,” and in “a variety of ways.” There of course is a “Standards and Guidelines for Developing a Collection of Evidence” they have created to steer the schools through the understanding of exactly what does and what does not constitute sufficient “evidence.” In other words, meeting this “standard” has no “standard” meaning. It is not a standard at all.

Finally, to get a CIM, a student must complete a Career Related Learning Experience, CRLE. This takes idiocy to a new level.

The CRLE calls for every student to go “beyond the classroom” for a learning experience. The schools are supposed to partner with local businesses, non profits, community groups and others to create “internships, structured work experience (paid or unpaid), clinical practicums, and mentorships.”

Now, this isn’t supposed to be just some kind of menial job type of thing. No, the CRLE must be relate to the student’s personal interests and goals, and must be “structured around learning goals with identified outcomes.”

So, supposedly the schools are going to have to arrange and monitor hundreds and hundreds of such internships, each with their own goals, assessments, learning objectives, etc? Is this realistic? Have the bureaucrats even considered the logistical implications of such a mandate? Are local businesses clamoring for teen babysitting projects?

Are we to believe that school staff would have the time to design an individual learning goals and assessment plan for every single student’s CRLE? Remember – the outcomes have to be identified and assessed!

The ODE has spent millions developing this phase of Vera’s reform, and it is just about ready for implementation. Last session we got the date pushed back to 2008, but the State Board of Education formally adopted the structure just a few months ago.

They are serious about this, and we can’t let it happen.