C2020 THEME

Evaluation utilization: achieving our potential?

Since the original Michael Quinn Patton’s Utilization-Focused Evaluation in 1978, evaluation utilization has become the ultimate value for many evaluators world-wide: for them, evaluation information has no purpose if it does not get used. But over the last 40 years, the very notion of evaluation utilization has become more complicated and its dynamics, more complex: there is talk of the utilization of individual evaluations, of evaluation as a process, of evaluative thinking, of evaluation insights, etc. We hear evaluation managers craving for their evaluations to be useful and used, where historically this was taken for granted. There is still confusion as to what constitutes a quality evaluation and whether it is defined by its process, its product, its use, or some other criteria. Moreover, evaluation is only one source of information potential users have access to – and not necessarily the most credible or actionable one. Also at issue is who gets to define what utilization evaluations are expected to have and how the needs, the expertise, and the interests of beneficiaries and disenfranchised groups are used to judge an intervention.

The 2020 Canadian Evaluation Society (CES) Conference will explore the world of evaluation utilization through three strands by first positioning it (what is it? is it a priority?), then by achieving it (how is it achieved? how do we know it has been achieved?), and by questioning it (whose utilization?).

Positioning Utilization

This strand investigates the nature of evaluation and evaluation utilization, its various forms and intensities, as well as its measurement. What is evaluation utilization? What different types of uses are there (Kusters identifies instrumental use, conceptual use, process use, symbolic use, relational use, value use, and external use)? Does pursuing some types of utilization prevent from achieving other types? Are some forms of uses more effective than others at supporting the common good? Can evaluation and evaluations be misused; if so, how and why does this occur (Alkin and King are inspirations)? What can be done to reduce the risk of misuse? How far have we come in assessing utilization: how do we observe, describe, and measure use? Does the value placed on utilization negatively affect how evaluations are designed, executed, and reported? Is it possible to conduct a quality evaluation without consideration for use (otherwise stated, is utilization a necessary condition of evaluation quality)?

 

Achieving Utilization

This strand examines the mechanisms at play in achieving evaluation utilization. What are the factors that determine whether or not an evaluation or evaluation information or an evaluation function, is put to maximum use? Are these factors context-dependent: are they the same across organizations, cultures, geography, languages, etc.? Which combinations of factors are more likely to maximize use and in which context? What is the role of evaluation capacity in evaluation use (give a read to Cousins and others)? What lessons can be drawn from the literature and practice on the use of information in decision-making and behavioural change? Does the flexibility given to federal departments and agencies in the 2016 Policy on Results help or hinder subsequent utilization? Has the increased emphasis on examining diversity within evaluation (e.g., SGBA+ analysis) enhanced use? Has an increased focus on performance measurement by programs increased the value of evaluations and their utilization or obscured the real aim of self-criticism?

Questioning Utilization

This strand addresses the subversive aspects of evaluation utilization. Who defines what utilization is sought for a given evaluation or for evaluation practice as a whole? Who determines what gets evaluated and how? Is evaluation use systematically reflective of the interests of those with resources to commission evaluations? Is the valuing of utilization a representation of the power structure in organizations and in society where the powerful may emphasize self-serving use over inquisitive knowledge? What is the responsibility of evaluators themselves in promoting use, if any: are we seditious agents of change or servile information gatherers? How can disenfranchised groups become part of the evaluation entreprise, and contribute to and benefit from evaluation utilization? Whose perspective defines what is legitimate use and questionable misuse are?