"The Plain Dealer's use of selective benchmarking results in misleading conclusions"

Submitted by Norm Roulet on Tue, 05/26/2009 - 15:27.

Cuyahoga County Auditor Office in Collinwoord

I was glad to see printed in the Sunday May 24, 2009, Cleveland Plain Dealer "Cuyahoga County staffing, services unfairly characterized in Plain Dealer series", by James McCafferty, Cuyahoga County's administrator. There he challenges several PD analyses of Cuyahoga County government performance, writing "The Plain Dealer's use of selective benchmarking results in misleading conclusions", which he claims are "disingenuous and intellectually dishonest".

I've spent several decades providing professional benchmarking services to some of the world's largest corporations and completely agree - the PD does not provide a good analysis here, it is not "apples to apples", as they say, and so it is harmful to the community.

I had intended to write something quite similar here about this issue, myself. I was glad to see McCafferty speak-up for his administration, and with some authority.

Bill Callahan, in is Cleveland Diary, had also observed the poor analyses by the PD and recently wrote "The Plain Dealer’s big “analytical” comparison of county agency spending (Cuyahoga County agencies have more workers, less productivity than others in Ohio, Plain Dealer analysis finds) is such a misleading, misdirected mess that it’s hard to know where to start."

After four years on the Greater Cleveland Lead Advisory Council, which works with Cuyahoga County on lead poisoning eradication, I know the county has many benchmarks in place measuring performance of their programs and initiatives - the work I have done with the county was analyzed under contract by Cleveland State. It is very valuable to benchmark aspects of government, over time and against best in class, and I would encourage expansion of benchmarking as a tool for continuous improvement in government, always.

But benchmarking is not the same as building a spreadsheet, and the PD does the practice disservice by suggesting their analyses are good science, thorough and unbiased. They best stay out of the competitive analytics businesses and stick to public relations... unless their editors feel negative campaigning against public confidence in County government is some form of public relations service.

The header of the day is a Cuyahoga County Auditor's neighborhood office in North Collinwood I just happened to come by, Sunday. It seems Cuyahoga County has many department and service outreaches like this across the region... is this good or bad? What is the value?

The county has a huge and complex scope and footprint. Such complexity - and the quality of services that result - drive cost and workload to determine value of government. To attempt to make the understanding of this more simple than that - to skip steps in measuring costs and benefits - leads to completely invalid results.

I'd love to see good results, but the PD can't deliver that - they do not have the resources.

AuditorOfficeCollinwoodPanLogo.jpg62.43 KB
CollinwoodAuditorSign650.JPG214.88 KB

Benchmarking is the act of

Benchmarking is the act of taking a measurement and then changing the process to see if the measurement was then affected, and how, by that change. Comparing per capita/unit costs between different counties is not benchmarking. If the factors are related to per capita variables within a comparison county then they can be weighted, but I am not to sure that the five counties are in fact that different. Before you say they are different then you must provide the data that in in fact demonstrates the statistical variations. To say they are different is subjective.

If the PD’s analysis is not valid then what variables or variations between counties would cause that, because if it is specific, and then if checked, and then if no significant variation exists, it would be in fact an excuse.

The bench marks are now set, please make them better and do not compromise quality of service while doing so. Any analysis on the average wait/delivery time for services? None of this is comprehensive, is it. I would like to say it is a start, but it could be just the beginning and the end of it.

If the method of measuring performance is fully defined, then we could run models to see if in fact a regional system will produce better results.

Wikipedia has an excellent write up on business benchmarking

Wikipedia has an excellent write up on business benchmarking, as developed at Rank Xerox and grandfathered by Robert Camp... http://en.wikipedia.org/wiki/Benchmarking

I do comparative best practice benchmarking, which offers outstanding value. To do this at the Cuyahoga County level, a group of counties would each need to make serious high level and organization-wide commitments to be benchmarking partners, for a period of many years, and consultants would work with county employees to determine what processes would be benchmarked, based on specific tools, data, definitions and methodologies developed for this purpose.

An effective methodology would explore quality of service, often requiring evaluation of customer satisfaction - internal and external data collection instruments must be developed and deployed across all benchmarking participants - all this and more to capture one cycle of data for evaluation. The greatest value is gained repeating the process over time. In the process, differentiators like complexity, scope of services, Service Level Agreements, customer variability, localization factors and such must be taken into consideration and data perhaps normalized. Best performers are identified and their best practices are analyzed - case studies are developed - demonstrated best practices may then offer opportunity for others.

The objective of such comparative best practice benchmarking is to identify measurable world class and best in class performers and the practices enabling best outcomes that may benefit other participants in the collaboration - it is a remarkable and valuable enabler of continuous total quality improvement, but an immense amount of work and not as simple as it sounds.

Disrupt IT

Very good follow-up, there

Very good follow-up, there are dangers in lay-people making very broad and general assumptions. Being higher than the average does not mean you are necessarily the highest over all. Comparing to an average is or can be inherently biased. Is there a statistical variation across all measured values? If the highest cost is not the most proficient then its wasteful. It’s interesting stuff, adding technology also adds costs. A single clerk working with one phone with a set of file cabinets cost less, but is not very proficient. I think it is good that the PD rattles the cage, but the responses from the peanut gallery can be unnerving and perhaps why things often happen behind closed doors.

Massage the data

  Any way you want to and you can claim benchmarks.  It doesn't take a rocket scientist to figure out we are not especially well represented by any and all Cuyahoga County administrations. 

Are there hardworking, decent county workers, especially, on the frontlines?  You bet.  But, if any one wants to fault the PD for this not especially revealing or hard-hitting analysis, then fault them for not  "benchmarking" personal income and liveability indicators like youth programming, headstart/daycare providers, good schools and access to higher education.  Our so-called leaders have failed us miserably.

Lots has been done to measure lead data

Lots has been done to measure the internal effectiveness of all the subcommitees of the Greater Cleveland Lead Advisory Council, which spends public funds, and of the outcomes of the spending of those funds, and the data is very specific and at times scientific, e.g. # of lead poisoned children by zipcode. So I know what the County does around this is well measured, analyzed and reported, and I'm pleased by lots of good outcomes - I believe there will be a report on this at our annual meeting, next month (I'll post this to realneo shortly). Far from finished, but showing measurable progress and some innovative practices.

I do not directly analyze anything about the county so I can't say where we may do better -

I have some ideas on how we may do better, though, and I'm letting the county and everyone else know.

Disrupt IT

Having knowledge and or an

Having knowledge and or an intuitive understanding of what the differences are between inferential and referential. Then seeing the value within the consolidation and standardizations of the uses of software and technology. Information systems and combined technology budgets and standard processes, a big cost saver. The county is and has repeatedly said the they want a county data center. That is very intelligent government and if that center is set up correctly it could support regionalism. Separate what is administration from what is political.

Change is a process and change to the process is reengineering and it has protocol. However what is new and different can be and will be revolutionary. The only rules are the rules of numbers, what is optimum and why.

What's the point here?

What's your point Norm--that the PD shouldn't bother to analyze Cuyahoga County administration by performance standards?  

Do you believe that the analysis reached the wrong conclusions?  Are you impressed by something I am missing?  Is some analysis better than no analysis?

Wrong analysis is far worse than no analysis

There is nothing worse than wrongful promotion of wrong data and wrong analyses, as the Plain Dealer does regularly to support their interests. People become misinformed in the most devious way... through supposed science. When Foundation leaders back up the bad analyses, as did the head of the Gund Foundation, in the case of the PD analyses of Cuyahoga government, it makes the Foundation party to the deception. Did the Foundation fund the analyses... who really did the analyses... better not to use such mind control.

Should we do good analyses of government? Absolutely. But it will not come through a three month part time analysis or the dark corridors of our foundations.

I'm driving such analyses here, and so are others like yourself who want honest data, analyses, and best practices rather than mind control.

Settle for nothing less!

Disrupt IT

Nearly 75,000 citizens -- 62 percent of the vote -- favored...

Here, in the PD's spinning of the latest politically charged issue... prosecution of the city for the violation of the city charter by a series of mayors, allowing them to bestow jobs upon who they chose without the call for a civil service exam, or open, fair hiring practices... "Nearly 75,000 citizens -- 62 percent of the vote -- favored the issue"... the "issue" being a charter amendment to wipe the slate clean for everyone.... which the PD spins as having good public support...

Is 75,000 Clevelanders good public support?

Who spun those issues to get those 75,000 good citizens to vote as they did?

Who paid for those votes? Who did the PR? Who did the polling? Who did the legal?

Disrupt IT

Real NEO Data and Benchmarks for the PD to analysze

From Environmental Health Watch, on real regional outcomes based on real science, reflecting performance of Cuyahoga County and other government agencies... we may drill down from here.

Cleveland Childhood Lead Poisoning Rates - 2007

  • Based on the CDC blood-lead level-of-concern (10 mcg/dl), 961 children in Cleveland were identified as lead-poisoned, 6% of those tested in 2007.
  • Based on the Cleveland/Cuyahoga County blood-lead level-of-awareness (5 mcg/dl), 3,322 children in Cleveland were identified as lead-poisoned, 22% of those tested in 2007.  
  • Only 28% of children were tested, so most children hurt by lead are never identified. Of the 72% of children who were not tested, no doubt many were also lead-poisoned in 2007.
  • In several Cleveland neighborhoods, a third or more of children tested at 5 or above. For a 2006 breakdown by census tract within neighborhoods see this chart.

Cuyahoga County Childhood Lead Poisoning Rates - 2004-2007

  • Based on the CDC blood-lead level-of-concern (10 mcg/dl), in 2007, 1,176 children in the Cuyahoga County were identified as lead-poisoned, 5% of those tested.
  • Based on the Cleveland/Cuyahoga County blood-lead level-of-awareness (5 mcg/dl), 4,070 children in the Cuyahoga County were identified as lead-poisoned, 16% of those tested in 2007.

Disrupt IT

“That is why Cuyahoga

“That is why Cuyahoga County citizens deserve a response to The Plain Dealer's recent series on staffing and services (May 10-12). In general, the series uses oversimplified, inaccurate and incomplete data to make comparisons to other counties. In at least one instance, The Plain Dealer's numbers are simply wrong. Furthermore, the series omitted the fact that Cuyahoga County spends less per capita than Franklin or Hamilton counties, despite these glaring realities: Cuyahoga County has more residents in poverty, more seniors, more job losses and more foreclosed homes than any other Ohio county.

PD Report Summary

Summary page two  

Which number are “simply wrong” Are the Franklin County Sheriffs’ Numbers? They are as stated the glaring outlier.

The whole thing is rather charged is the prosecutor over worked or efficient? Is the county engineer efficient or are all the bridges falling apart?


Cuyahoga is the largest and spends the least per capita? What are the sources of the Budgets? Who gets the most intergovernmental funding? 

per capita spending by county:

Hamilton $1,455.84

Cuyahoga $1,140.25

Montgomery $1,729.11

Franklin $1,440.22

Lucas $1,397.16

I believe that the funding supports poverty and does little to eradicate it. The report shows that we spend less dealing with our high crime and decrepit infrastructure. If Cuyahoga ,as I suspect, receives the most intergovernmental funding then that means our tax revenues are less. That with the highest rate in sales tax.

We have a county that is what dependent on poverty and intergovernmental funding? The industry of need?


Nearly 75,000 votes

That's simply a fact -- not spin, Norm. I don't make any judgment whether that's "good" support or "bad" support. But of those who turned out, 62 percent favored the issue.

Since a facet to the city's argument is that the judge is overruling the will of the voters, it's only fair to report the factual number of voters and the percentage. I devoted equal time to the judge/civil servants' side, and the lead of the story dealt with how the judge ruled the charter amendment unconstitutional. It was a decision that potentially impacts scores of city employees and the city budget, which affects all taxpayers. Whether you agree or disagree with the merits of the case; whether the city is right or wrong in this matter, at least that much is true.

My goal, as always, is to be an honest broker.

Two games in town

 From my vantage point, I see two games played on taxpayers in C-town: cronyism/patronage/contract awards and accelerating poverty/and theft of property through the CDC/corporate-backed oligarchy in NEO.  I don't care how these two games get exposed and shut down.  I am just glad that the PD is finally daylighting these schemes.

The sun keep rising....

... and the PD keeps being delivered each morning

.... but PD and "daylighting" just don't fit in the same universe.

Disrupt IT



yogi and guy http://www.nationalwardogsmonument.org/

Almost half of Cuyahoga

Almost half of Cuyahoga County deputies didn't take civil service exam

Why would anyone want to be in politics if they cannot hire their friends and family?   

How well do the municipalities respect the Civil Service laws? Cleveland does not what about the others?