wpeA.jpg (1735 bytes)

M. H. West & Co., Inc.
A Planning and Consulting Company

700 East Main Street, Suite 904, Richmond, VA 23218    TEL: 804-782-1938    TOLL FREE: 1-888-WEST904    FAX: 804-782-9771

 

RSAT LINKS

Linking Residential Substance Abuse Treatment Professionals to Resources and Information

Sept 00, VOL. I, No. 7 A  Publication By M. H. West & Co., Inc.

for Virginia’s RSAT Grant Programs

© M. H. West & Co., Inc.

PROGRAM OUTCOME ANALYSIS - MEASURING SUCCESS

Outcome analysis is important in providing information on the successful utilization of the citizen’s dollars, as allocated by government. The development of an effective outcome analysis is an issue that has surfaced during visits to the RSAT grant programs. Some states require mandatory outcome analysis by statute. Virginia does not, though the Department of Mental Health, Mental Resources and Substance Abuse Services is implementing the first phase of its statewide performance outcome system. This issue of "Links" discusses one method that may be useful in developing outcome analysis for substance abuse programs. The discussion is based on the NIDA publication "Measuring and Improving Costs, Cost-Effectiveness, and Cost-Benefit for Substance Abuse Treatment Programs". The guide may be downloaded from www.drugabuse.gov

Thomas L. Wilkinson.

 

In order to measure the effectiveness of the use of public (local, state and federal) dollars for substance abuse treatment of offenders, and demonstrate responsible stewardship of this public money, an accurate and thorough outcome analysis is warranted. Obtaining accurate outcome data requires an effective method to track the re-socialization of ex-offenders. This issue of Links provides data on a method developed by NIDA that may be useful in evaluating the effectiveness of substance abuse programs.

Program administrators, funding sources and the general public want to know: How effective is the program at helping graduates to re-assimilate into society and remain drug and arrest free, and at what cost was this obtained?

 

Conducting a Cost Analysis

In designing an outcome analysis, the type and amount of cost data to be collected, the method(s) to be used to collect the data, and the frequency of collection should be determined.. Some basic cost categories are:

  • Personnel (social worker supervisors, regional clinical supervisors, managers clinical social workers, counselors and administrative personnel, etc.)

  • Volunteer personnel

  • Both direct and indirect space used

  • Furniture and equipment and supplies (including any donated materials)

  • Transportation

  • Communication services (telephone, email, security, etc.)

  • Vendor services (drug tests, accounting, security, etc.)

  • Insurance (such as professional liability) and finance

  • The development of a cost spread sheet will allow for the routine collection of this data.

  • Other factors that might be considered are:

  • Ensuring that volunteer personnel and donated equipment or supplies are priced at the cost associated with having to purchase these services.

  • Stating indirect costs as a percentage of direct costs. Examples of indirect costs are overhead, advertising costs, and shared costs with the facility, such as utilities. Indirect costs can be allocated as a percentage cost per client.

  • Determining the amount of program space used and its cost. Remember to include the cost of furniture and accessories as well as the portion of utilities associated with the space.

  • Calculating the usage cost by multiplying the amount of time the space is used by the cost of the space.

  • Including the percentage of program time utilized by inmates who were discharged from the program, when determining the number of clients served. It is tempting to exclude from evaluations of treatment outcomes data, inmates who were put out of the program before completion.

  • Breaking down total costs into cost per client This can be done by dividing the total cost of the program by the number of clients served. This is usually done at a point in time, that is the total cost and client count on a given day

Cost analysis provides a basis for developing the cost-effectiveness (relationship between program cost and program effectiveness) and cost-benefit analysis (measurement of both costs and outcomes in monetary terms) of the program.

 

Measuring Cost-Effectiveness

In measuring the cost-effectiveness of the program, it is important to determine the measurement parameters to use. Some common ones are the following:

  • Drug-free days

  • Employed days

  • Number of recidivists within first year, two years, etc.

  • Number of arrests within first year, two years, etc.

Outcomes after treatment can be: compared with before treatment experiences, or with a control group who has not received the treatment. However it is important to note that:

  • before and after comparisons of outcome may exaggerate the impact of treatment, since treatment often is started because a client’s behavior became progressively more severe, which is what got him /her into jail/prison in the first place; and

  • in comparing to a control group, one must be sure that the control group norm is similar to the treatment group norm. For instance, criminal behavior tends to reduce with age, so ages should be similar.

 

Linking Costs to Outcomes

The most complete and useful cost-outcome analysis includes a detailed cost analysis of the treatment procedures and inmate processes conducted that brought about the outcomes. These analyzes necessitate the performance at the individual client level. This level of detail gives a clearer picture of cost versus outcome but is more complex and expensive to generate. If one has the resources to calculate the costs for each client, the sum of the cost of all clients is probably the most accurate cost data

Graphs conveniently show the relationship between costs and outcomes. Microsoft Excel, Quattro Pro or Lotus 123 are all excellent for this purpose. As an example, the cost of treatment can be compared to drug-free days upon release.

Methods for analyzing outcomes are the following:

  • The cost-outcome ratio for clients is obtained by dividing the average total cost per client by one outcome measure or by a composite of all the outcome measures made.

  • A second method would be to divide the total cost of treatment by one outcome measure or by a composite of all the outcome measures made.

 

Analyzing Cost-Benefit

There are several potential measures of the cost-benefit of the program.

 

  • New income generated. This is a weak measure because often employment for ex-offenders who are also substance abusers is very difficult.

  • Cost savings to society or taxpayers.. These cost savings include: funds otherwise spent in the illicit economy of drugs; criminal justice services not required; and social and health services no longer needed.

Measures of these are the following:

  • Decreases in clients involved in driving while intoxicated or driving under the influence.

  • Decreases in client’s families who sought counseling.

  • Reduced crime and use of the judicial and criminology systems.

  • Decrease in involvement in accidents.

  • Increased participation in aftercare programs.

  • Reduction in the use of social services.

The reduction in criminal activity following substance abuse treatment may not produce a corresponding reduction in actual costs to society. Although costs to citizens drop in direct proportion to reductions in criminal acts perpetrated on those citizens, public expenses for criminal justice services may not decline in a similar manner. Typically, police courts and other components of the criminal justice services are on a fixed budget, while the need for criminal justice services greatly surpasses the ability to deliver those services. For this reason, the impact of substance abuse treatment on criminal justice resources saved because of a reduction in crimes committed by former substance abusers may be diverted to other criminal justice services. The entire budget for criminal services probably will still be spent.

Similar problems may occur when cost savings benefits are measured for reduced health, mental health, and future drug treatment services. Because resources in these services typically are very limited, the actual reduction in expenditures may not be as much as might be expected from the reduction in patient use of services.

Nevertheless, transforming effectiveness findings into estimated cost savings still may have considerable value for a program evaluator. In particular, cost savings estimates can show the magnitude of criminal justice and treatment resources that are now available to help other drug abusers who previously could not be helped because of budget restrictions.

The following tables show ways to calculate types of costs and potential cost savings for benefit analysis, along with potential benefits that can be obtained.

 

 

Table 1

Types of Costs and Potential Cost Savings

Effectiveness Measure

Effectiveness-Benefit Transformation

Benefit Measure

Criminal acts not performed


Thefts at $_ /misdemeanor

$_ /felony

Assaults at $ _

Savings to potential victims due to income loss avoided, property not damaged or lost, and health and mental health services not needed
Drugs not purchased
Opiates at $ _to $ _/day

Cocaine and crack at $_to $_ /day

Other at $ _to $_ /day

Money not spent
on drug purchases
Criminal justice services not used
Arrests at $ _/arrest

Jail at $_ /day

Prosecution at $ _/day

Expense of criminal justice
services avoided
Drug treatments no longer needed
$_ per client day for mixture of
treatments provided
Cost of drug treatment
no longer needed
Welfare payments not provided $ _per client per day for mixture of treatments provided Amount of welfare
payments provided
Disability payments not made $ _Per client per day in disability payments
Size of disability
payments not made

Health services not used

Sum of health care cost used for 6-12 months before treatment and 6-12 months after treatment
Cost of health
services not used

 

 

 

Table 2

Potential Benefits Obtained

Employment (licit) Income earned from licit sources
Entrepreneurship (licit) New income (profit) from enterprise
Income taxes paid on licit income Amount of Federal, State and local taxes paid on licit income
Increased productivity in an existing job Increased profit for employer, company, and sole proprietorship

 

Cost-benefit analysis measures whether the outcomes of a program are worth the costs by:

  • Measuring outcomes in the same units. This is usually cost in dollars.

  • Seeing whether the value of outcomes exceeds the value of costs (by subtracting total costs from total benefits, which is called net benefit).

Other ways that benefits can be expressed are in:

  • Present-value benefits - a way to show the immediate positive outcomes from an extended future period. Consult the publication or a statistics text for the method of calculation.

  • Time to return on investment - the time at which the investment equals monetary outcomes. The time it takes for benefits to exceed costs is a useful measure in showing results. This is particularly effective in graph form.

Some potential things to look out for when developing or analyzing a cost-benefit analysis are:

  • being careful that the correct assumptions are made from the ratios presented.

  • placing overemphasis on the monetary outcomes without consideration of the non-monetary and anecdotal outcomes of the program .

 

Comparing Programs and Components

 

An important component in outcome analysis is being able to compare between programs or components to determine which is the most effective in terms of cost, outcomes or both. There are a number of ways to do this. The Fishman table below shows how to apply cost and benefit in a comparison of two programs.

 

Table 3 - Fishman Table

 

Cost

Outcomes

A has better outcomes than B A and B have similar outcomes A has worse outcomes than B
A has lower costs than B Choose A Choose A Uncertain
A and B have similar costs Choose A Choose either Choose B
A has higher costs than B Uncertain Choose B Choose B

The table above does not effectively answer the question: How does one decide when one program costs less than another, or when one program has better outcomes than another?

Statistical tests can show how big a difference is and if the difference is real, that is if it is statistically significant. Ratios that can help are: cost-benefit, cost-effectiveness and cost-effectiveness vs. cost-benefit. Cost-effectiveness ratios are useful in decision making for comparing dollar outcomes between programs. If one program has a cost-per-drug-free day higher than another, it can be considered less effective. The cost-per-drug-free day has a number of useful characteristics. A day free of drugs is something concrete and understandable for most people. The challenge of this can be appreciated so this becomes a tangible measure. Measures such as this that people understand are useful in describing program outcomes. Comparing cost-effectiveness and cost-benefit ratios together to form a composite performance ratio that will take into account performance outcome and cost in one number is also useful.

Graphing can help to show comparative performance. Cost-outcome graphs are visual pictures of program performance. For greater detail on graphing consult the NIDA outcome analysis guide or books on statistical methods.

Outcome analysis is important to capture program success for use in informing legislators, government agencies and the general public about how the program is benefiting society. Providing regular reports on program progress is beneficial in keeping those who need to know informed. Tailoring the reports to fit the needs of the recipient is useful in getting maximum results from the analysis made.

 

For more information on outcome analysis, please contact Tom Wilkinson at 804.782.1938 or at mhwestcon@aol.com.

 

     Main Page----Criminal Justice Services----Services & Programs----Publications 

 
Send an e-mail to emailed.gif (14893 bytes)mhwestcon@aol.com for more information on our services.
M. H. West & Co., Inc.
700 East Main Street, Suite 904, P. O. Box 548 - Richmond, Virginia 23218-0548
TEL 804.782.1938  FAX 804.782.9771
© 2007
M. H. West & Co., Inc..