Another report urges changing API

A report this week from a Washington think tank bolsters Senate President pro Tem Darrell Steinberg’s call for significantly revising the state’s primary accountability measure, the Academic Performance Index. Now, if Gov. Jerry Brown would only read it…

“Ready by Design: A College and Career Ready Agenda for California,” published by Education Sector, recommends that the API shift focus from students’ performance on standardized tests to measures of readiness for college and careers, such as high school graduation rates, results of Advanced Placement tests, and percentages of students needing remediation in college. Account-EdSectorStudyCover062012That’s essentially what Steinberg’s bill, SB 1458, would do without specifying what measures would be included, and that is what his bill last year, SB 547, would have done, had Gov. Jerry Brown not vetoed it with a snarky message sharply critical of quantitative gauges of school achievement.

The report concluded that additional measures would not be a panacea, “but an outcomes-oriented API would at least measure and reinforce what’s most important: graduating students from high school with the knowledge and skills to succeed in higher education and a career,” wrote co-authors Anne Hyslop, a policy analyst with Education Sector, and Bill Tucker, deputy director, policy development at the Bill & Melinda Gates Foundation. The report was funded by the James Irvine Foundation, a big promoter and funder of an initiative to combine college readiness and career technical education, called linked learning.

This is the second report from Ed Sector in a month on the California API. Last week, I wrote about “Academic Growth over Time,” in which author Richard Lee Colvin, a former executive director of Ed Sector, recommends Los Angeles Unified’s alternative way to calculate student results on standardized tests. The reports need not be mutually exclusive; recommendations from both could be combined.

When establishing the three-digit API index in 1999, the Legislature implied that it would be adding a range of measures, but never did. Instead, the API is based on California Standards Tests results, primarily English language arts and math, plus results on the high school exit exam, which are not indicators of preparation for life beyond high school. But by adopting Common Core standards two years ago, the state embraced the goal of aligning high school achievement with college and career expectations. The API needs to change to reflect this, the Ed  Sector report says.

In his veto message of SB547 last year, Brown said that changing API now, when facing the challenges of new Common Core curriculum, texts, and tests, “doesn’t make sense.”

Sharply rebutting that, the authors wrote, “In fact, what doesn’t make sense is implementing new standards, tests, and curriculum that are aligned with college and career readiness while continuing to evaluate school performance based on an entirely different goal. More important, Brown’s continued opposition to CALPADS, his refusal to apply for federal funds to improve the state’s data collection, and his veto of the API redesign, only serve to maintain the very status quo that he repeatedly condemns.”

While college acceptance rates are indicators of readiness, the report recommends adding evidence of post-high school achievement to the API, such as college remediation rates, percentage of students who return to college after one year, and the percentage of students who enter the military or who go straight to work after high school. Unlike states like Florida, which track students after high school, California has incomplete, voluntarily collected data – hence the criticism of Brown for forgoing chances for federal database money.

Underutilized EAP measure

California does have one measure of readiness that other states consider a model: the Early Assessment Program, a series of questions created by the California State University and administered as a supplement to CSTs taken by high school juniors. But the authors said that “while touted as a model for those developing Common Core assessments, its respect outshines its influence.” The University of California and most community colleges don’t use results of EAP, and most high schools aren’t using the followup courses in math and expository writing that would enable high school seniors to bypass remediation in college.

Quoting a 2006 survey that found 10 percent of high school teachers said their students graduated not ready for college, compared with 44 percent of college faculty who said students arrived unprepared, the report cited “mismatched expectations on both sides.” Including college readiness measures in the API would encourage high schools, colleges and business leaders to work closer to create common expectations and share expertise and knowledge. The report cited promising examples: collaboration between a San Diego high school and the Grossmont-Cuyamaca Community College District to redesign high school English courses; and the extensive agreements between Long Beach Unified and the local community college and CSU, Long Beach.

Calling SB 1458 “my highest priority,” Steinberg said, “I really believe this bill would bring about a change in cultures. It would create incentives to link education with needs of economy in a fundamental way.”

“The Administration is fine with the API the way it is. I disagree and want to limit the importance of testing,” he said.

The bill, which has passed the Senate, will be heard in the Assembly Education Committee on June 27. Steinberg said he hoped to negotiate with Brown on the measures in the bill. It includes the possibility of including school inspections, which Brown raised in his veto message and mentioned in his State of the State message in January.

Board of Ed draws QEIA line in sand

Administrators from Stockton Unified School District went home about $3 million poorer yesterday. They had traveled to Sacramento to appeal to the State Board of Education for waivers from the academic requirements of the Quality Education Investment Act (QEIA). All five of the district’s K-8 schools receiving QEIA funds had failed to meet their Academic Performance Index targets, and that meant being expelled from the program at the end of this school year.

The principals argued their cases, some more compelling than others. Monroe had undergone an entire change in leadership, Grunsky had two special day classes added after becoming a QEIA school, Roosevelt and Van Buren experienced high teacher turnover, and Nightingale, a new charter, needed more time to prove itself.

It didn’t take long for the State Board to decide the schools’ fate. Denied.  Not just for Stockton, but also for another five schools in other districts.

“We took a bullet today,” a school principal was overheard saying to a colleague as they left the meeting room.

This month’s meeting marked the second time QEIA schools that missed their API marks had come to the Board seeking relief; and it was the second time that Board members rebuffed them. The Board has been liberal in approving waivers for going a little over class size reduction and a little under the teacher experience index, but for academic achievement, well, it’s a bit like Tevye’s ethical choice in Fiddler on the Roof: “On the other hand… No… there is no other hand.”

“They accepted the money. This was a voluntary grant, which they accepted under clear and explicit conditions and, therefore, in following the intent of the law they need to meet those conditions,” said Board President Michael Kirst after the meeting.

The entire board appears to have arrived at a consensus on this issue.  “We seem to have drawn a collective line with regard to the API,” said board member Carl Cohn.  “Even though circumstances have changed dramatically, you still have to have some central tenet of a program like this and, for me, it is the API. ”

But Stockton school officials find the reasoning inconsistent.  If the goal is to raise student achievement, why take away the money that’s providing additional teachers, resources, counselors and professional development necessary to help students learn.  Van Buren principal Ione Ringen said without QEIA funds she’ll lose more than a quarter of her 32 teachers, boosting class size from 20 to 32.

“Losing this is going to negatively impact our students,” said Ringen.  “We come from high poverty areas with high numbers of second language learners, and having that opportunity to be in a smaller class has allowed them to have more individualized education.”

QEIA, for a quick refresher, was created to settle a lawsuit brought by the California Teachers Association against Gov. Schwarzenegger for failing to keep his promise to repay school districts and community colleges $2 billion “borrowed” from Proposition 98 in 2004-05 to help the state get through a budget crisis.

Under the agreement, schools in the lowest academic levels were eligible to apply for funds to improve achievement by maintaining class size reduction, hiring more counselors, providing more directed professional development, having more highly qualified teachers than the district average, and exceeding their API growth targets averaged over three years.

State Board of Ed action on QEIA waivers since January 2009. (Source:  California Dept. of Ed). Click to enlarge.
State Board of Ed action on QEIA waivers since January 2009. (Source: California Dept. of Ed). Click to enlarge.

Since QEIA began, the State Board has approved 86 waivers and rejected six on non-academic grounds, and denied 16 for not meeting API goals.  As of now, nearly 140 QEIA schools out of 474 have failed to meet at least one requirement.  [Click here for chart] .

Department of Education staff warned the board on Thursday to expect waiver requests to mount.  At least fifty more are on the docket for the July meeting propelled in part by the financial crisis that’s forced schools to lay off teachers and scale back class size reduction.

One waiver request expected to be reconsidered at the May board meeting could break the de facto API denial rule.  Over the past few years, thousands of Iraqi refugees have resettled in eastern San Diego County, within the boundaries of El Cajon Valley High School.

The school population swelled by more than 700 students, nearly all English learners.  El Cajon redirected some of its QEIA funds to organize community and social service organizations to help the students and their families adjust to life in America.  The school’s  API score dropped.

Department of Education staff recommended denial of the waiver, but the board postponed its vote until the next meeting, and asked the staff to try to peel off the refugees’ test scores to see how the school would have done without them.  Regardless of those results, most board members suggested that this could be one of those difficult and exceptional situations that has no precedent.

“It seems to me that that as a state we have an obligation to ensure that our districts welcome these youngsters and make sure that they’re we’ll supported,” said Cohn.  “This is markedly different from any other waiver request that’s been before us.”

Steinberg’s API alternative

Rebuffed by Gov. Jerry Brown last year, Senate President pro Tem Darrell Steinberg is back with another academic accountability bill, this time giving the governor lots of latitude to help redefine how to measure schools’ performance. SB 1458 needs to be vague, because, at this point, no one but Brown professes to know what he has in mind.

Recognizing that the Academic Performance Index, based predominantly on English and math standardized test results, was too narrow a gauge, Steinberg last year proposed replacing the API with an Education Quality Index that would have included other indexes, such as dropout rates, the need for remediation in college, success with career technical education programs, and graduation rates. Standardized tests would have counted no more than 40 percent of the EQI in high school. Steinberg and key supporter Superintendent of Public Instruction Tom Torlakson built an impressive coalition of supporters – business and civil rights groups, career and technical education groups, charter schools, the state PTA, and early childhood education advocates.

But in his veto message of SB547, Brown criticized the continued reliance on quantitative measures. “SB 547 would add more things to measure, but it is doubtful that it would actually improve our schools. Adding more speedometers to a broken car won’t turn it into a high-performance machine.”

In the new bill, Steinberg would retain the 40 percent maximum use of the API, and would instruct Torlakson to expand the use of science and history tests within it. As for the remaining 60 percent, SB 1458 would allow Torlakson and the State Board of Education to incorporate another idea that Brown mentioned in his veto message and State of the State message: school inspections or visitations to measure the quality of learning and instruction not measured by standardized tests. Brown has not clarified if he is talking about a corps of outside inspectors, as is used in England by the Office for Standards in Education, Children’s Services and Skills (Ofsted) or a less formal system of intradistrict inspections.

The State Board, led by Brown advisor Michael Kirst, plans to make the adoption of new accountability measures a priority this year. At his instigation, the San Francisco-based nonprofit, nonpartisan research organization WestEd has surveyed other states’ accountability systems and analyzed all of the data that the state collects. Its report has not yet been released.

The timing is right for a new accountability system. Brown proposes to give near-total control over spending and budget decisions to local districts this year, as part of his school finance reforms. It will become imperative to  create better ways to measure whether schools are providing a rich environment for learning, spending dollars effectively on students who have been targeted for extra money, and preparing students well for post high school jobs and colleges.

A case study for NCLB waiver

A couple hundred children sitting cross-legged covered the floor of the multipurpose room at Oak Ridge Elementary School in Sacramento. Behind them, parents, grandparents, and siblings filled rows of metal folding chairs, while teachers stood beside their students. The room was abuzz with excitement as principal Doug Huscher bounded onto the stage and led everyone in a cheer.

“When I say ‘Oak Ridge,’ you say ‘Feel the pride,'” shouted Huscher. Three times he called and they responded.

It was the warm-up to the school’s first-ever awards ceremony for student performance on the California Standards Test (CST). Teachers presented more than 200 Olympic-style medals to their students: bronze for moving up at least one level on the exam, silver for scoring in the proficient range, and gold for advanced.

It was also a celebration for the school itself. Two years ago, when it was in the bottom 5 percent in the state for academic achievement, Oak Ridge, along with five other low-performing schools in the Sacramento City Unified School District, applied for a piece of the $316 million in federal money allotted to California through the School Improvement Grant program. They lost.

So superintendent Jonathan Raymond launched his own reform, the Priority Schools program. He installed new principals and many new teachers in the six schools, gave them additional funds from Title I and the American Reinvestment and Recovery Act, told them to come up with a school improvement plan, and held them accountable.

Raymond sees this as a prototype for education reform efforts and that’s why he’s pushing the State Board of Education to seek a waiver under No Child Left Behind (NCLB), in which the federal government would give districts and schools flexibility and not hold them to the rigid models of NCLB and other federally-funded reforms.  A waiver would also allow low-income schools that receive Title I funds to use that money as they see fit.  So far the State Board has not been persuaded to apply.  Raymond said if they did, it would also solve another cruel irony related to Oak Ridge; after all its work and achievement on California’s standards, it’s still considered a failing school under NCLB.

Academic and behavioral changes at Priority Schools. (Source:  Sac City Unified School District). Click to enlarge.
Academic and behavioral changes at Priority Schools. (Source: Sac City Unified School District). Click to enlarge.

After Huscher’s first year as principal of Oak Ridge, its ranking on California’s Academic Performance Index, or API, soared by 82 points, from 658 in 2009-10 to 740 last year. State Superintendent of Public Instruction Tom Torlakson recognized the accomplishment last August, by holding his news conference to announce statewide API results at Oak Ridge.

Given those plaudits, Huscher was understandably frustrated when the school again failed to meet the federal benchmarks for Adequate Yearly Progress (AYP) under No Child Left Behind. Out of 25 measurements, Oak Ridge fell short in one: Asian students who were English learners didn’t score high enough on the English language arts exam. Oak Ridge wound up in year five of program improvement, opening it to a number of severe penalties.

“After all this effort, you have this amazing increase in API, and the feds come in and say, ‘Guess what?  You didn’t make it,'” said Huscher.

The disconnect between state triumph and federal failure isn’t unusual in this era of No Child Left Behind. Of the 3,890 California schools in Program Improvement, 476 met their API targets. What’s different today is that the U.S. Department of Education realizes that NCLB is a flawed law, and is offering waivers to states to provide relief from sanctions, including those that are psychological.

“The punitive elements of NCLB – this public labeling of schools that are working hard to improve learning – are damaging and destructive,” said Raymond. A waiver, he added, “would go a long way in helping to remove stigma and repair the reputations of many schools.”

But State Superintendent Torlakson has so far convinced the State Board of Education not to apply, warning that the requirements in exchange for a waiver are too costly. California is one of ten states that haven’t submitted either formal requests for waivers or letters stating they intend to apply.

Oak Ridge an example of waiver flexibility

Under pressure from local superintendents, the State Board is going to reconsider the issue at its next meeting in March.

Sac City’s Raymond said schools like Oak Ridge are case studies of what’s possible when they’re given latitude to be creative and develop improvement plans based on their specific student populations instead of the prescriptive remedies and sanctions of federal education officials. “We had that, and it didn’t get us anywhere,” he said.

Through the Priority Schools program, Oak Ridge principal Huscher has been able to hire a training specialist who meets weekly with teachers from each grade to analyze data from student work, design interventions and lessons based on the data, model those lessons for the teachers, and work in their classrooms if necessary.

“The first year I was here it didn’t seem like there was very much collaboration between grade levels,” said third grade teacher Kelly Toomey, one of four teachers who remained on staff after it became a priority school. “I think the most important change that I’ve seen is that the focus and the culture of the school across the board with parents and students is academics.”

During a visit to her class earlier this year, Toomey had nearly every student engaged in a lesson on the difference between expository and narrative writing, using two books about penguins, one nonfiction and one fiction.

She broke into song when Tacky the penguin sang “how many toes does a fish have?” sending the kids into giggling fits; she paused after reading the word “odd” to make sure the students understood it; and she prodded them to think deeper.

“So what could we say about narratives?” Toomey asked one of her students. “It’s make believe,” he answered. “But was your personal narrative story you wrote make believe?” she prompted. “No,” he said, “it was very true.” “It was very true,” she agreed. “This is odd, like Tacky.”

Strengthening the home school connection

Teachers are required to make home visits, and Toomey said she’s noticed more parent involvement as a result. They ask questions about academics and homework, and they come to school more often.

Anthony Bookhamer’s grandmother said it was important for him to have his teacher see where he lives. Anthony won two gold medals at the awards ceremony, one for math and one for English language arts.  After returning to his spot on the floor, he looked through the door to his right, into the hallway where his grandmother, Lenna Tryon, watched from the seat of her walker. Anthony waved enthusiastically and raised the medallions off his chest to show her.

“The school’s so good for him,” said Tryon. “They know he’s got special things; he has to have water, he has to wear a hat, and he can’t play more than ten minutes or be out in the sun more than ten minutes…and they set him up in the front so he can see better; his eyesight is pretty bad. He’s done remarkable and just everything’s come back to him.”

About two years earlier, Anthony was in an apartment fire that killed his mother and left him with burns over 70 percent of his body. His hands, legs, back, and stomach are scarred, he lost several fingers, and he has a bald spot on the back of his head where the flames scorched him. Anthony spent five months in the hospital and went to live with his grandmother.

Before Huscher came to Oak Ridge, it was a different place, said Tryon. There were no awards for the kids, she never received any communication telling her what was going on, and she didn’t feel welcome at the school. Now they call her a few times a week to let her know how Anthony’s doing and what’s going on; they have family game nights and monthly parents’ meetings.

Sac City has been adding more schools to its Priority School program, and Raymond is looking for ways to keep it going and sustainable. A waiver would help in that as well. Relief from NCLB sanctions would free up millions of dollars that districts with program improvement schools are required to set aside for private tutoring companies that have no accountability.  It amounts to $2.4 million for Sac City.

“We sort of look back and we chuckle now because we didn’t get the [School Improvement] grant for perhaps a variety of reasons, none of which made sense to us,” said Raymond.  “But we said the heck with it, we’re going forward anyway, we’ve got a good plan, we believe in it.”

Brown vetoes API alternative

Calling it “yet another siren song of school reform,” Gov. Jerry Brown has vetoed a bill that would have expanded the state’s accountability system to include measures other than standardized tests.

SB 547, the top education priority of Senate President pro Tem Darrell Steinberg, was one of 15 education-related bills that Brown killed on Saturday, the day before the deadline for acting on legislation before him. Among the others: SB 185, a direct challenge of Proposition 209’s ban on considering race and ethnicity in admitting students to CSU and UC; and AB 203, modifying the Parent Trigger law.

In a sharp, two-page veto message of SB 547, Brown mocked “academic ‘experts,’ ” backed by “editorialists and academics alike,” who have “subjugated California to unceasing pedagogical change and experimentation.” He singled out the “current fashion” of collecting “endless quantitative data … to distinguish the educational ‘good’ from the educational ‘bad.’ ” Instead, Brown indicated that he favors a “focus on quality” instead of quantity – with measures such as “good character or love of learning,” as well as “excitement and creativity.”

As to how to do this: “What about a system that relies on locally convened panels to visit schools, observe teachers, interview students and examine student work? Such a system wouldn’t produce an API number, but it could improve the quality of our schools.”

Steinberg and Superintendent of Public Instruction Tom Torlakson, a sponsor, had rounded up widespread and diverse support for SB 547 from business groups, some advocates for low-income children, the career technical education (CTE) community, and much of the education establishment – the PTA, school boards, and administrators associations. (The California Teachers Association, which will cheer Brown’s anti-testing rhetoric, took no position on the bill.) Even an organization representing gifted students signed on.

Replace with Education Quality Index

Some supporters, Steinberg included, fundamentally disagree with Brown over the use of data to measure student and school performance. Others acknowledge that standards-based reforms and standardized tests, as demanded by the Legislature and the federal government, are here to stay. All agree that the current system, basing a school’s Academic Performance Index mostly on annual math and English language arts tests, narrowed the curriculum in many schools and created perverse incentives to focus on testing.

SB 547 would have replaced API with an EQI, an Education Quality Index, that would have added more indices, particularly in high school. Measurements could have included dropout rates, the need for remediation in college, success with career technical education programs, and graduation rates. Standardized tests would have counted no more than 40 percent in high school, no less than 40 percent in K-8, as determined by the state Department of Education and the State Board of Education. Backers of the current system questioned whether the EPI would be too squishy. Brown took the opposite view ­– that it would have demanded more of the same, hard data.

In his veto message, he also criticized the timing, taking effect at the same time that the state was switching to Common Core standards in math and English language arts, with their own set of demands. The combination would “add significant costs and confusion,” Brown wrote.

But Steinberg disagrees, noting that the transition to the Common Core standards, with a focus on college and career readiness, is the right time to change the accountability system to reflect that priority.

“It’s a fine idea that the governor wants qualitative pieces, but that does not change the fact that our high schools are not focused on the economy and what we expect young people to do when they graduate from high school,” Steinberg said in an interview.

“I disagree with his view on data, which can show what works and what doesn’t; that is what taxpayers want with their money. What we are doing (with SB 547) is not negating quality measures, just trying to improve quantitative measures.”

Steinberg said he would meet with Brown soon to create a bill in 2012 that fixes “a flawed system that has negative consequences for children and schools.”

(Readers: Is Brown a visionary or a policy Luddite? What do you think?)

More applied learning meeting A-G

Brown did sign two other bills that Steinberg sponsored to encourage more hands-on learning in high school. SB 611 will encode in statute the new UC Curriculum Integration Institute, which brings together CTE and core academic teachers, along with UC professors, to design innovative courses, blending applied learning, that satisfy A-G course requirements for admission to UC and CSU. The Institute has created a half-dozen so far with limited funding; with SB 611 in hand, Steinberg says he will approach foundations to underwrite the effort for hundreds of additional courses.

SB 612 complements SB 611 by reauthorizing the California Subject Matter Projects, which provide teacher training and development for courses created by the Institute and related courses.

Other vetoes

SB 185: In his veto message, Brown said that he actually agreed with the intent of the bill, which would have allowed CSU and UC to consider race, gender and  ethnicity  when considering undergraduate and graduate admissions, and that he wrote briefs backing the position when he was attorney general.

But the courts, not the Legislature, must determine the limits of Prop 209. Passing the bill, sponsored by Sen. Ed Hernandez (D-West Covina), “will just encourage the 209 advocates to file more costly and confusing lawsuits.”

AB 203: The veto of a bill dealing with the “Parent Trigger” law was a surprise, since the sponsor, Julia Brownley (D-Santa Monica), who chairs the Assembly Education Committee, had gone to great lengths to get Parent Revolution, the chief proponents of the law, and skeptics to agree to the language. It clarified pieces of the parent empowerment law, which the Legislature passed in a hurry in late 2009. The law permits a majority of parents at a low-performing school to petition for a wholesale change, such as a conversion to or takeover by a charter school.

But Brown said that the State Board has spent a full year writing regulations covering the petition process and these should be allowed to work before changing the law.

In a statement expressing her disappointment, Brownley said the bill “could have reduced potential litigation over the law’s ambiguities” by clarifying aspects of the signature process. Ben Austin, executive director of Parent Revolution, credited Brownley for collaborating and listening to parents with his group, then added, “But I do think the Governor’s veto sends a strong signal that it’s time to stop tinkering and start implementing the Parent Trigger.”

Brown skeptical of key ed bill

Gov. Jerry Brown has warned lawmakers that his veto pen will flow freely over the next three weeks. Among bills on the threatened list is potentially the most far-reaching K-12 education legislation before him – a bill that would significantly shift the state’s accountability system away from its concentration on standardized tests. SB 547 is also a priority of its author, Senate President pro Tem Darrell Steinberg.

Acknowledging that the bill may be in trouble, Steinberg said Friday that Brown “has expressed some doubts” about it, while not precluding the possibility that he might sign the bill, in several discussions they have had.

Brown, too, has expressed a dislike of the reliance on standardized tests by the state and particularly the federal government with the No Child Left Behind law to measure the success of students and teachers. So one would think that he would be simpatico with Steinberg on SB 547, which would create an Education Quality Index, or EQI,  to replace the Academic Performance Index, or API, with new indices to downplay standardized tests.

Brown’s apparent objection isn’t about the bill’s cost but instead about uncertainties over a possible demand for new data. Brown has a visceral distrust of statewide data systems and use of data in general. He killed money for a statewide database on teachers – CALTIDES – and wanted to delete additional federal funding for the state’s database on students, CALPADS, though the Legislature reinserted it into the budget.

SB 547 would create new indices measuring a broader range of student achievement, such as career and college readiness, accomplishment in areas outside of core subjects, and high school graduation and dropout rates. For high schools, standardized tests would be a maximum of 40 percent of the new EQI; for middle and elementary schools, standardized tests would comprise a minimum of 40 percent of the EQI.

“This bill is consistent with his (Brown’s) philosophy of getting away from test scores,” Steinberg said.  And with Congress deadlocked over the reauthorization of the Elementary and Secondary Education Act, the formal name for No Child Left Behind, California should set its own priorities and “lead by example,” Steinberg said.

Superintendent of Public Instruction Tom Torlakson and an advisory committee would develop the EQI, but the State Board of Education, whose members are Brown appointees, would have to approve it. That would give Brown control over the indices, Steinberg said. Rather than veto the bill, Brown could sign it with a message signaling the changes he would want to the bill next year, he said. (He also credited Brown for spending a lot of time and thought on the bill.)

Meanwhile, Steinberg has been campaigning to raise visibility for the bill; last week, he and Torlakson held a press conference in Los Angeles with Torlakson, U.C. Regent George Kieffer, and representatives from business and civil rights groups, the state PTA, and early childhood education advocates to call on Brown to sign it.

The bill has substantial support from diverse groups (see list at the end of Steinberg’s fact sheet), representing business and manufacturing, career technical education, gifted students, charter schools, school administrators, and school boards. The California Teachers Association hasn’t taken a position on SB 547; the California Association of School Counselors and the California Business Educators Association are backing it. Steinberg is hoping other groups will speak up between now and Oct. 9, the last day for Brown to sign or veto bills.

John can be reached at

Expansion of test for special ed students distorts latest API results

The 2011 California Academic Performance Index (API) results were released yesterday, and revealed slow but steady gains compared to previous years’ results. However, changes in California’s Standardized Testing and Reporting (STAR) program have led to artificial increases in API gains over the past four years. Sorting out how much of the “good news” announced yesterday is based on true academic achievement gain made by California kids and how much may be due to changes in the STAR program is not a trivial issue.

The numbers. If we record statewide API results for the last four years by grade span, including the results announced today, we find the following (see API chart):

API results (click to enlarge)
API results (click to enlarge)

Overall, the gains for 2011 appear to be marginally less than the reasonably steady gains recorded in 2008, 2009, and 2010. The multiyear gain scores from 2007 to 2011 are more noteworthy: 47 points for grades 2-6 and 58 points for grades 7-8. The question is: Are these multiyear gains true achievement gains?

The program changes. In 2008, California began introducing the California Modified Assessments (CMAs) to the STAR program for selected Special Education students. The CMAs have been phased in over four years – for grades 3-5 in 2008, grades 6-8 in 2009, and grades 9-11 over two years in 2010 and 2011. The CMAs are easier tests than the mainstream STAR California Standards Tests (CSTs), designed for Special Education students who score very low on the mainstream CSTs. Special Education students who scored Far Below Basic or Below Basic on a STAR CST in the previous year are eligible to take a CMA, with final determination for which test to take left to the student’s Individualized Education Program team. The CMAs were initiated in response to a federal flexibility program, and were to be targeted for no more than 2 percent of overall enrollment, or roughly 20 percent of Special Education enrollment.

As documented in a TOPed post, initiation of the CMAs over the past four years has affected STAR statewide summary data reported by the State Superintendent. By systematically eliminating students who score Below Basic or Far Below Basic from the STAR CST program, the percentages of students who scored Proficient or Advanced on CSTs have been artificially increased by an estimated 26 percent over four years. In addition to the problem of accuracy for summative data for STAR, the TOPed post also documented the fact that CMA usage has increased far more rapidly than anticipated and has far outstripped the anticipated 2 percent of overall enrollment or 20 percent of Special Education enrollment.

For grades 3-8, where CMAs have now been used for several years, CMAs were administered to 5.2 percent  of the total number of students tested in grades 3-8 in 2011, and more CMAs than CSTs were administered to Special Education students. Finally, when one looks at CMA usage by district, one finds a wide discrepancy among districts in the percentage of Special Education students administered the CMAs – from less than 20 percent for some districts to almost 70 percent for other good-sized districts. For a sampling of wide differences, see the chart for Santa Clara County.

in Santa Clara County, grades 3-8. The CAPA entries reflect estimates based on Levels II-IV only. The Santa Clara Co numbers are taken from STAR database and do not reflect column sums. The numbers are taken from the STAR database released 8/15/11. Chart by Doug McRae (click to enlarge)
The numbers in the chart are for districts in Santa Clara County, grades 3-8. The CAPA entries reflect estimates based on Levels II-IV only. The Santa Clara Co numbers are taken from STAR database and do not reflect column sums. The numbers are taken from the STAR database released 8/15/11. Chart by Doug McRae (click to enlarge)

The crux of the issue for the 2011 API results is that the introduction of easier tests for a significant portion of Special Education students generates inflated API multiyear scores. If the 2 percent limit originally envisioned had been strictly applied to API calculations, the artificial increases in API scores would likely not be a major factor.  However, with the CMA usage running 2 to 3 times higher than anticipated, the artificial increases in API scores are quite notable. Also, CMA now becomes a potential way for local districts to game the API system, testing more Special Education students with CMA to artificially increase their API scores.

The Big Picture. I discussed the larger policy implications for the introduction of CMAs to California’s STAR statewide assessment system in a TopED commentary posted earlier this year. In that commentary, I note the issue is not the idea of a modified assessment – I agree that CMAs yield better data for selected Special Education students than counterpart CSTs.  The issues raised here are (1) How should CMA scores be treated for API calculations? and (2) Are CMAs being overused and/or abused by local districts to artificially raise API scores?

The API Inflation Effect. How much are the API scores inflated?  It is relatively easy to estimate this; a year ago I computed the API inflation effect for the 2010 API scores; it was in the 35 to 40 percent range for grades 3-8. With the same methodology used a year ago, I’ve computed the inflation effect for the 2011 API scores due to the introduction of the CMAs over the past four years. Since the CMA performance levels for all tests grades 9-11 have not been set as yet, it is possible to estimate the inflation effect only for grades 3-8 at this time. The results of the 2011 API inflation effect  are (see CMA inflation chart):

CMA inflation rate (click to enlarge).
CMA inflation rate (click to enlarge).

The bottom line here is that the statewide API increases reported for the past four years for grades 2-6 have been inflated by 42.4 percent, and the statewide API increases reported for the past three years for grades 7-8 have been inflated by 34.4 percent. Stated another way, more than a quarter of the statewide gains claimed by the SPI/CDE over the past 3-4 years have been artificial gains due to changes in the STAR program (i.e., introduction of CMAs) rather than true achievement gains.

Counterarguments/Remedies. When I raised the issue of inflated API statewide gain results last year, CDE staff presented two counterarguments:

  • Initially, CDE staff claimed 2010 API results were not inflated; instead, API scores in previous years were deflated due to the administration of inappropriate tests to selected Special Education students. In other words, the previous years’ CST scores were not valid while the current year’s CMA scores are valid. However, even if this argument is true (and I do not challenge it from an individual student point of view), the gains in API scores over multiple years will be inflated unless either the “deflated” API scores from previous years or the “inflated” API scores from this year are adjusted so that there are apples-to-apples comparisons to calculate gains.
  • CDE staff claimed that appropriate adjustments are made for each year’s Base API to Growth API calculations, to take into account the replacement of scores from the more rigorous CSTs by scores from the less rigorous CMAs. However, this analysis does not address API 1-yr Base-to-Growth results; rather it addresses multiyear API trend data. Other technical adjustments made during the API calculation process also do not address the distortion that CMAs have introduced to multiyear API gain data.

There are relatively simple remedies for the problem that California is reporting inflated API multiyear gain data. Adjusting CMA performance level scores downward for API calculations is one solution that has precedent in the API system. However, remedies need to be vetted by California’s API advisory committee and then presented to the State Board of Education for approval before they can be executed. So far, neither CDE staff nor the SPI nor the SBE has shown the leadership needed to fully vet the issues involved with the introduction of CMAs to California’s statewide STAR assessment system and to consider appropriate remedies.

Doug McRae is a retired educational measurement specialist living in Monterey. In his 40 years in the K-12 testing business, he has served as an educational testing company executive in charge of design and development of K-12 tests widely used across the US, as well as an adviser on the initial design and development of California’s STAR assessment system. He has a Ph.D. in Quantitative Psychology from the University of North Carolina, Chapel Hill.