«Анализ и оценивание государственных программ и отраслевых политик» Обязательная учебная дисциплина для 3-го курса факультета ГиМУ ГУ-ВШЭ (4 и 5 модуль 2006/07 уч. года) ^ Аннотации статей / распределение по группам и докладчикам(update 06/04/07, 09.00 МСК) № п/п Наименование статьи /имя файла Abstracts / кол-во страниц без обложки и списка литературы Примерная дата выступления ^ ФИО:Самозапись/ Распределение 1-ый блок – этапы политико-управленческого цикла ^ Тема №2. Триада понятий «политическое». Циклы Policy making и Policy cycle Turning Swift Policy-makingTSPM.pdf Most member states of the European Union (EU) have some difficulty in transposing EU directives. Despite the obligation to comply with EU law, member states are often slow to adopt national policies implementing directives. In this paper I analyse this problem by focusing on the coordination of transposition in the domestic policy arena. Coordination is approached as a game in which one or more higher-level players decide on policy when lower-level players are unable to make a decision. Based on the model developed in the paper, lower-level players sometimes appear to have discretion in shaping the policy transposing a directive. Furthermore, if a single player coordinates the transposition process, the implementing policy differs from the policy specified by the directive. However, a decision-making process with more than one higher-level player can result in deadlock, leading to a literal transposition of a directive. Moreover, deadlock between the deciding players may delay the transposition process. Both mechanisms are illustrated by two cases of decision-making on EU directives in The Netherlands: the cocoa and chocolate products directive and the laying hens directive. The analysis shows that the framework developed in this paper contributes to the understanding of transposition./ с. 23 7 мая Волкова Ю., 391 ^ An Emerging Framework for Including Different Types of Evidence inSystematic Reviews for Public PolicyEmerging.pdf Evidence-informed policy and practice call on research addressing a broad range of research questions: evaluating the need for, and development, implementation, acceptability and effectiveness of interventions. Synthesizing this evidence requires methods that integrate the findings from diverse study designs. This article reports the development of a new model of research synthesis for this purpose. On completion of a series of substantive reviews, methodological reflections addressed: the interrelationship between review questions, relevant theory and values within the review process; methodological similarities and differences with more conventional reviews of effectiveness; the added value in terms of conclusions and specific recommendations; and the relevance to public policy. / с. 16 7 мая Кривошеев А., 391 ^ Dynamics of power in contemporary mediapolicy-makingMedia PM.pdf One of the most remarkable aspects of the controversy surrounding The Federal Communication Commission (FCC) review of US media ownership rules in 2003 was the passionate participation of the public. While hundreds of thousands of critical responses flooded into the offices of the FCC and congressional staff, public meetings across the US were packed out with citizens largely hostile to the FCC’s proposals to loosen ownership regulations and the prospect of increased consolidation. Such levels of involvement were unprecedented given the public’s traditional reluctance to get involved in issues concerning the ‘shape’, rather than the content, of the media. This reluctance is partly due to a perception of the policymaking process as remote and overly technical, the domain of engineers, lawyers and civil servants, and partly because the media have failed to register as a key part of public policy-making as compared to health, education or taxation. / c. 16 12 мая Рамазанов Р. ^ SOCIAL BENCHMARKING, POLICY-MAKING ANDTHE INSTRUMENTS OF NEW GOVERNANCE IN THE EUPM EU.pdf The European Union has established an “open method of coordination” (OMC) among the Member States, as a means of pursuing economic and employment growth, coupled with greater social cohesion. The paper analyses the assumptions underlying this strategy, the manner of its operation and its implications for policy learning and for governance. It argues that the OMC has in substantial measure been inspired by the experience of EMU, but that this is a perilous paradigm on which to base social benchmarking and the OMC. Second, it argues that the dynamic development of the OMC will depend crucially on the extent of public involvement in scrutinizing the policy areas which political leaders choose for benchmarking, the best practices which they identify in other Member States, and the efforts which they then make to improve national performance. The conclusion is that the OMC is likely to have substantial implications for domestic and European governance, but that these could be very different, depending on the manner of its implementation./ с. 16 14 мая Изотов М., 392 ^ Policy-making in the EUPM in the EU.pdf Комплексное рассмотрение политики в Европейском союзе / с. 24 14 мая Савин С., 392 ^ Тема № 3. Восприятие проблемы (1-я стадия политико-управленческого цикла) Issue Networks and the Executive EstablishmentIN&EE.doc Подробное описание политики и управления в США. . / с. 12 12 мая Ковальчук А., 393 ^ The Dynamics of Issue Expansion: Cases from Disability Rights and Fetal Research ControversyDIE.pdf Статья посвящена роли общественного мнения в политике. / с. 19 7 мая Степанова Е., 391 ^ Policy Issue Network and the Public Policy Cycle: A Structural-Functional Framework for Public AdministrationPIN.pdf В статье рассмотрены вопросы соотношения политики и управления. / c. 14 14 мая Османова В., 392 ^ Тема №4. Формирование повестки (2-я стадия политико-управленческого цикла) Agenda Building as a Comparative Political Processcobb ross ross.pdf Сравнительное описание формирования повестки в странах мира. / c. 13 12 мая Слабкевич Г., 393 ^ How Public Policy is MadePPM.pdf Статья посвящена общетеоретическим основам формирования повестки с точки зрения места этого процесса в политике/ с. 9 14 мая Маскалева О.*, 392 ^ Is Greener Whiter? Voluntary EnvironmentalPerformance of Western Ski Areasrivera deleon.pdf This study analyzes the initial implementation of the Sustainable Slopes Program a voluntary environmental initiative established by the U.S. National Ski Areas Association in partnership with federal and state government agencies. Our findings indicate that participation of western ski areas in the Sustainable Slopes Program is related to institutional pressures in the form of enhanced federal oversight and higher state environmental demands exerted by state agencies, local environmental groups and public opinion. The analysis also suggests that, despite these institutional pressures, participant ski areas appear to be correlated with lower third-party environmental performance ratings. This behavior seems to reflect the lack of specific institutional mechanisms to prevent opportunism in the current design of the Sustainable Slopes Program. That is, the program does not involve specific environmental standards, lacks third-party oversight, and does not have sanctions for poor performance / с.18. 7 мая Рыбакова О., 391 Тема №5. Формулирование политики (3-я стадия политико-управленческого цикла) ^ Is There Life After Policy Streams, AdvocacyCoalitions, and Punctuations: Using EvolutionaryTheory to Explain Policy Change?john.pdf This article reviews the current state of public policy theory to find out if researchers are ready to readdress the research agenda set by the classic works of Baumgartner and Jones (1993), Kingdon (1984) and Sabatier and Jenkins-Smith (1993). After reviewing the influences of institutional, rational choice, network, socio-economic and ideational approaches, the article pays tribute to the policy streams, punctuated equilibrium and policy advocacy coalition frameworks whilst also suggesting that future theory and research could identify more precisely the causal mechanisms driving policy change. The article argues that evolutionary theory may usefully uncover the micro-level processes at work, particularly as some the three frameworks refer to dymamic models and methods. After reviewing some evolutionary game theory and the study of memes, the article suggests that the benefits of evolutionary theory in extending policy theories need to be balanced by its limitations. / c.16. 16 мая Зайцев Ю., 391 ^ The Science of Muddling ThroughSMT.pdf Статья посвящена теории формулирования политики / c.14 21 мая Каплинская Д., 392 ^ The Impact of Agenda Conflict on Policy Formulation and Implementation: A Case of Gun Controlvizzard.pdf Вопросы формулирования политики рассматриваются на примере обращения с оружием и различными ограничениями в этой области / с.7 19 мая Карташков П., 393 ^ Nonincremental Policy Change: Lessons from Michigan’s Medicaid Managed Care Initiativeweissert goggin 2002.pdf Статья посвящена вопросам формулирования и реализации политики / с.9 19 мая Карташков П., 393 ^ Тема №6. Методы принятия решений STATISTICAL MODELS FOR CAUSATION What Inferential Leverage Do They Provide?Statistical Models for Causation.pdf Experiments offer more reliable evidence on causation than observational studies, which is not to gainsay the contribution to knowledge from observation. Experiments should be analyzed as experiments, not as observational studies. A simple comparison of rates might be just the right tool, with little value added by “sophisticated” models. This article discusses current models for causation, as applied to experimental and observational data. The intention-to-treat principle and the effect of treatment on the treated will also be discussed. Flaws in per protocol and treatment-received estimates will be demonstrated. / c.21 16 мая Варламова Д., 391 ^ An exploratory analysis of treatment completion and client and organizational factors using hierarchical linear modelingAn exploratory analysis of treatment.pdf Data from the Alcohol and Drug Services Study (ADSS) are used to analyze the structure and operation of the substance abuse treatment industry in the United States. Published literature contains little systematic empirical analysis of the interaction between organizational characteristics and treatment outcomes. This paper addresses that deficit. It develops and tests a hierarchical linear model (HLM) to address questions about the empirical relationship between treatment inputs (industry costs, types and use of counseling and medical personnel, diagnosis mix, patient demographics, and the nature and level of services used in substance abuse treatment), and patient outcomes (retention and treatment completion rates). The paper adds to the literature by demonstrating a direct and statistically significant link between treatment completion and the organizational and staffing structure of the treatment setting. Related reimbursement issues, questions for future analysis, and limitations of the ADSS for this analysis are discussed. / c.15 21 мая Перцев Я. * ^ A Subjective and Objective Integrated Approach to Determine Attribute Weights9802.pdf This paper proposes an integrated approach to determine attribute weights in the multiple attribute decision making problems. The approach makes use of the subjective information provided by a decision maker and the objective information to form a two-objective programming model. Thus the resultant attribute weights and rankings of alternatives reflect both the subjective considerations of a decision maker and the objective information. An example is used to illustrate the applicability of the proposed approach. / c.15 19 мая Нехай В., 393 ^ KNOWLEDGE ACQUISITION ANDEXPLANATION FOR MULTI-ATTRIBUTEDECISION MAKINGAvignon88.pdf A decision making approach which combines multi-attribute decision making techniques with expert systems is described in the paper. In this approach, knowledge about a particular decision making problem is represented in the form of tree-structured criteria and decision rules. Decision making is supported by an expert system shell. In the paper, the discussion is mainly concentrated on the part of the shell which deals with interactive knowledge acquisition and knowledge explanation. The corresponding algorithms are given and discussed. / c.18 16 мая Усачева О., 391 ^ Decision making and knowledge management in inquiring organizations: toward a new decision-making paradigm for DSSCourtney_2001.pdf В статье рассмотрены вопросы использования систем поддержки принятия решений / c.21 16 мая Сыромятников А., 391 ^ Regulatory Focus and Strategic Inclinations:Promotion and Prevention in Decision-Makingcrowehiggins1997.pdf A promotion focus is concerned with advancement, growth, and accomplishment, whereas a prevention focus is concerned with security, safety, and responsibility. We hypothesized that the promotion focus inclination is to insure hits and insure against errors of omission, whereas the prevention focus inclination is to insure correct rejections and insure against errors of commission. This hypothesis yielded three predictions: (a) when individuals work on a difficult task or have just experienced failure, those in a promotion focus should perform better, and those in a prevention focus should quit more readily; (b) when individuals work on a task where generating any number of alternatives is correct, those in a promotion focus should generate more distinct alternatives, and those in a prevention focus should be more repetitive; and (c) when individuals work on a signal detection task that requires them to decide whether they did or did not detect a signal, those in a promotion focus should have a “risky” response bias, and those in a prevention focus should have a “conservative” response bias and take more time to respond. These predictions were supported in two framing studies in which regulatory focus was experimentally manipulated independent of valence. / c.15 21 мая Карякина Л, 392 ^ Water quality simulation modeling and uncertaintyanalysis for risk assessment and decision makingEcomod1994.pdf The usefulness of water quality simulation models for environmental management is explored with a focus on prediction uncertainty. Ecological risk and environmental analysis often involve scientific assessments that are highly uncertain. Still, environmental management decisions are being made, often with the support of a mathematical simulation model. In the area of pollutant transport and fate in surface waters, few of the extant simulation models have been rigorously evaluated. Limited observational data and limited scientific knowledge are often incompatible with the highly-detailed model structures of the large pollutant transport and fate models. Two examples are presented to illustrate data and knowledge weaknesses that are likely to undermine these large models for decision support. An alternative to comprehensive structured simulation models is proposed as a flexible approach to introduce science into the environmental risk assessment and decision making process. / c.18 21 мая Щербатова Э., 393 ^ Application of the Choquet Integral in Multicriteria Decision MakingGraRouColl2000.pdf In this paper, we introduce the Choquet integral as a general tool for dealing with multiple criteria decision making. After a theoretical exposition giving the fundamental basis of the methodology, practical problems are addressed, in particular the problem of determining the fuzzy measure. We give an example of application, with two different approaches, together with their comparison. / c.24 19 мая Пономарева Е., 393 ^ ARTIFICIAL NEURAL NETWORK MODELS FOR FORECASTING AND DECISION MAKINGHMOR93.pdf Some authors advocate artificial neural networks as a replacement for statistical forecasting and decision models; other authors are concerned that artificial neural networks might be oversold or just a fad. In this paper we review the literature comparing artificial neural networks and statistical models, particularly in regression-based forecasting, time series forecasting, and decision making. Our intention is to give a balanced assessment of the potential of artificial neural networks for forecasting and decision making models. We survey the literature and summarize several studies we have performed. Overall, the empirical studies find artificial neural networks comparable to their statistical counterparts. We note the need for using the many mathematical proofs underlying artificial neural networks to determine the best conditions for using artificial neural networks in the forecasting and decision making. / c.15 19 мая Шатова И., 392 ^ Тема №7. Реализация программ и политик (4-я стадия политико-управленческого цикла) From Knowing to DoingFrom Knowing to Doing.pdf The past decade has witnessed widespread interest in the development of policy and practice that is better informed by evidence. Enthusiasm has, however, been tempered by recognition of the difficulties of devising effective strategies to ensure that evidence is integrated into policy and utilized in practice. There is already a rich but diverse and widely dispersed literature that can be drawn upon to inform such strategies. This article offers a guide to this literature by focusing on six main interrelated concerns: (1) the types of knowledge relevant to understanding research utilization/evidence-based practice (RU/EBP) implementation; (2) the ways in which research knowledge is utilized; (3) models of the process of utilization; (4) the conceptual frameworks that enable us to understand the process of RU/EBP implementation; (5) the main ways of intervening to increase evidence uptake and the effectiveness of these; (6) different ways of conceptualizing what RU/EBP means in practice. / c.20 28 мая Балашова Е., 392 ^ Primary and Secondary Barriers to the Evaluation of ChangePSB.pdf For more than two decades it has been widely acknowledged within The management literature that the evaluation of change rarely takes place within organizations, yet little coherent explanation has been offered as to why this should be so. This article reports on research undertaken in two UK public sector organizations, to explore the reality of evaluation in the context of change, with the intention of identifying the barriers to evaluation that existed and the factors that created them. The analysis of the empirical evidence led to the identification of two distinct types of barrier to evaluation, labelled primary (factors that act against an evaluation being undertaken) and secondary (factors that arise during an evaluation process). Primary barriers were rooted in contextual factors contained in the organization’s history and culture; the background against which the change initiative took place. Secondary barriers relate directly to the choices and decisions involved in the evaluation process itself and therefore only become significant once a decision to undertake an evaluation has been made. In combination, they offer an explanation for the exclusion of the evaluation stage in many change initiatives. / c.18 23 мая Кулинич Н., 391 ^ Evaluating the Implementation ProcessEIP.pdf This article presents a specific analysis of programme-implementation evaluation with reference to European Social Fund (ESF) interventions. Although in recent years increasing attention has been paid to implementation processes, there is still a lack of adequate guidelines and methodologies to support this type of evaluation. Within the framework of the ESF programmes and of the Italian political-institutional context, this article analyses the main factors affecting the implementation process and considered relevant for achieving policy objectives. A case study of the Italian Objective 3 Community Support Framework process implementation evaluation is presented and the influence of three factors – actors, context and regulations – on outputs is described. / c.15 26 мая Богачева А., 393 ^ EVALUATION OF CURRICULA IN HIGHER EDUCATION Challenges for EvaluatorsEVALUATION OF CURRICULA.pdf Evaluation of teaching at universities is traditionally realized in terms of student ratings. Curriculum evaluation is rarely done in a systematic manner. More often, the emphasis is placed on a particular aspect, which is only of little help in terms of modifying education. A very prominent example is that of medical education. Here, evaluations of curricula primarily focus on new curricula by contrasting them to traditional ones. The article at hand deals with a different evaluation approach, in which five phases have to be considered and contrasting results to other teaching formats are not the main focus. In this article, the authors concentrate on the first phase (baseline evaluation) of the systematic evaluation of a medical curriculum. They describe several challenges of such an evaluation approach and illustrate the strategies used to overcome them. In addition, associated relevant empirical findings from this evaluation study are presented./ c.19 28 мая Линева Д., 392 ^ Evaluability Assessment: A Tool for Incorporating Evaluation in Social Change ProgrammesEvaluability.pdf In this article we discuss the evaluation of a particular type of health promotion programme where social change is a central theme of programme development, implementation and evaluation. Evaluation of social change programmes requires a politics of accountability and a utilization focus. These require a process of planning evaluations that mirrors the process of programme planning and implementation. Such an approach must be open to the possibility that the demands on evaluation research may change over the lifecycle of the programme. We propose an evaluability assessment framework as a participatory tool for planning evaluations that meet the need for credible evaluative accounts of social change and contribute to social change. This framework is a tool that can be used at any time to plan and review evaluations. / c.14 26 мая Родин Е., 393 ^ Reducing Anxiety and Resistance in Policy and Programme EvaluationsReducing.pdf Evaluators of an organization whose programmes or policies are under evaluation frequently encounter anxiety and resistance from the evaluees. Literature abounds with suggestions for developing collaborative interactions. Few studies, if any, show circumstances where the evaluees’ need for co-operation and support transcends the inherent anxiety and resistance and triggers for the evaluees’ desire for moral support and advice. An analysis of the socio-psychological factors involved in the circumstances described in this study may shed light on how the evaluator can develop the evaluation process to enhance the collaboration with the evaluees. Here we report and analyse three case histories of such evaluations to determine the critical features of the evaluations that made them collaborative rather than conflictual. The common issues raised shed light on practices that alleviate anxiety in the evaluation process. / c.19 23 мая Лободанова Д., 391 ^ 2-ой блок - оценивание Тема №8. Оценивание и мониторинг (5-я стадия политико-управленческого цикла) ^ Scale, Complexity and theRepresentation of Theories of ChangeSCR.pdf International aid agencies face major problems when attempting to evaluate their achievements. Activities, intended beneficiaries, partner institutions and social contexts are diverse because of their global scale. How can agencies’ ‘theories of change’ be adequately represented in summary forms that respect the complexity and diversity? This article, the first of two, looks at a range of types of change processes and how they can be represented. It starts with linear processes, traditionally represented via the Logical Framework, and ends with network processes. The solutions proposed are informed by three related cross-disciplinary theoretical perspectives: evolutionary theory; complex adaptive systems; and social network analysis. Examples draw on the author’s consultancy experience with international development aid programmes in Vietnam, Bangladesh, Cameroon and Burkina Faso. / c.20 30 мая Календарь Е., 391 ^ Evidence-based Policy: In Search of a MethodEvidence-based Policy.pdf Evaluation research is tortured by time constraints. The policy cycle revolves quicker than the research cycle, with the result that ‘real time’ evaluations often have little influence on policy making. As a result, the quest for evidence-based policy has turned increasingly to systematic reviews of the results of previous inquiries in the relevant policy domain. However, this shifting of the temporal frame for evaluation is in itself no guarantee of success. Evidence, whether new or old, never speaks for itself. Accordingly, there is debate about the best strategy of marshalling bygone research results into the policy process. This article joins the imbroglio by examining the logic of the two main strategies of systematic review: ‘meta-analysis’ and ‘narrative review’. Whilst they are often presented as diametrically opposed perspectives, this article argues that they share common limitations in their understanding of how to provide a template for impending policy decisions. This review provides the background for Part II of the article (to be published in the next issue, Evaluation 8[3]), which considers the merits of a new model for evidence-based policy, namely ‘realist synthesis’. / c.24 23 мая Хлопяк М., 391 ^ Desk Screening of Development Projects: Is It Effective?Desk Screening.pdf Donor agencies rely to varying degrees on desk reviews by headquarter staff for the selection, monitoring and evaluation of projects initiated by nongovernmental organizations (NGOs). If information provided in the various documents submitted by NGOs is inadequate or incomplete then the reliability of desk reviews as a management tool for allocation of project funding becomes questionable. A study of European Union funding of NGO projects compared findings from a desk evaluation of a structured sample of 30 projects with those from a field assessment of the same projects. The findings show that a field assessment on the basis of more complete information from a wider range of stakeholders leads to a markedly lower scoring on most of the standard criteria of project evaluation. An important policy implication is that ex ante screening, monitoring and evaluation on the basis of project documents are an ineffective management tool for donor agencies if not backed up by substantive follow-up in the field. This article places such management tools in the wider context of principal–agent relations. / c.18 2 июня Толстякова С., 393 ^ Meta-evaluation: Discovering What Works Best in Welfare ProvisionMetaevaluation.pdf Numerous programmes designed to encourage welfare recipients to move into work and off benefit have been evaluated in the United States. Many have randomly assigned potential participants into ‘experimental’ and ‘control’ groups to generate unbiased estimates of the effectiveness of the programmes. The results of the evaluations have been selectively influential in shaping policy developments on both sides of the Atlantic, but a thorough understanding of the diversity of experience has been lacking. Applying meta-analysis techniques to a specially constructed database of evaluations in over 50 US sites, this article reports on the first programme-level, systematic meta-evaluation of welfare-to-work programmes. The results confirm the superiority of approaches that prioritize immediate work over human-capital investment but reveal that caseload characteristics and local environment can be equally important as or even more important than programme design. The article concludes with a discussion of the potential and limitations of meta-evaluation. / c.23 23 мая Игнатьева М., 391 ^ Polity, Politics and Policy Evaluation in BelgiumPPP Evaluation.pdf This article examines the institutionalization of policy evaluation in Belgium. In comparison to other western democracies, policy evaluation is an underdeveloped tool of Belgian public governance. Among the explanatory factors of the laggard situation of Belgium, we focus on the all-pervasive phenomenon of partitocracy, on the relative weakness of Parliament vis-à-vis the government, and on the federalization process that is characteristic of the recent institutional evolution of the country. These three peculiarities of the Belgian polity and politics jeopardize the development of a mature evaluation culture. Finally, the empirical evidence for the Belgian case also calls into question several of the theoretical hypotheses that have been formulated to explain the diffusion of policy evaluation in various democracies. / c.19 26 мая Паскар А., 393 ^ Responsive Evaluation and Performance ManagementResponsive Evaluation.pdf Worldwide, governments are under increasing pressure to deliver results. There is general recognition of the importance of performance measurement and a results-oriented focus for effective public management. The shift from inputs to outputs and outcomes is accompanied by an increased use of performance indicators and policy targets. Critics point to the negative effects of static forms of performance-measuring and governance: simplification, resistance to change and a strategic use of indicators. They stress the need for an approach to governance that respects and is responsive to diversity and dynamism. Such an approach to public policy should recognize the importance of stakeholders opinions and motives and, hence, dialogue. Herein, policy evaluation can help by systematically questioning the validity of policy goals and performance indicators. Based upon recent experiences with results-oriented budgeting in the Netherlands, this article outlines a possible approach to a responsive, ‘verifying’ policy evaluation. Against the background of the traditional functions of evaluation in results-oriented management (assessing efficiency and effectiveness), the paradoxical nature of performance indicators is considered: on the one hand they are ‘frozen ambitions’, yet on the other they must facilitate dialogue and learning. Building on these attributes, a dynamic perspective on evaluation is put forward. In this, using information and insights from stakeholders, policy evaluation should explicitly question the validity of the policy objectives and performance indicators without compromising their value and significance altogether. In this way, the evaluator may fulfill the ambition of the ‘argumentative turn’ in policy evaluation: to help to raise the quality of dialogue and decision-making between stakeholders. / c.14 2 июня Пинхасов С. ^ Public Managers amidst AmbiguityPublic Managers.pdf Contemporary public management is characterized by a strong tendency to introduce performance measurement in order to reduce complexity. Public managers face two challenges when performing their work: uncertainty and ambiguity. Ambiguity is understood as the absence of or contradictory interpretations about what needs to, can and should be done, when and where. In this article we argue that the intensity and nature of ambiguity vary, depending on the public management setting. This has serious implications for the type of evaluation chosen. Performance measurement may be appropriate when ambiguity is relatively low, but it is difficult and potentially damaging in settings marked by a high degree of ambiguity. In these latter cases, evaluation approaches that acknowledge ambiguity through dialogue are more suitable. To structure this line of reasoning, we distinguish four public management settings (industrial, enforcing, professional and strategic) and relate this to different evaluation approaches. / c.20 30 мая Ерке Г., 391 ^ Contracting Out Municipal AuditingContracting.pdf This article evaluates the contracting out of municipal auditing. It draws upon the organizational economics (theory of the firm) tradition, specifically agency theory and transaction cost analysis. In order to systematically assess the effects of contracting out municipal auditing, a conceptual framework of municipal auditing is developed. Some empirical evidence is then discussed in relation to this framework. In general, enhanced competition and contracting out municipal activities seem to lead to an increase in both cost-efficiency and effectiveness. However, there are potential problems related to market contracting of municipal auditing, i.e. the prospect of reduced auditor independence, reduced audit quality and in particular less performance auditing. Finally, the article presents suggestions for future research, indicating the need for further empirical studies, especially on how the contracting out of municipal auditing affects auditor independence and performance auditing, and how mixed ownership/plural governance structures affect audit outcomes. / c.18 26 мая Гельманов П., 393 ^ The Evaluation of Policy Coherence for DevelopmentThe Evaluation of Policy Coherence.pdf The promotion of human welfare in the zones of turmoil and transition can no longer be evaluated simply by examining the degree to which aid programs and projects achieve their relevant objectives efficiently. With globalization, the rules of the game that govern the international economy have become pivotal. Yet, the policies that govern trade, investment, migration, knowledge flows, environmental protection and human security have escaped systematic evaluation. To help ascertain the impact of rich countries’ policies on the economic and social prospects of poor countries, evaluators must broaden the scope of their assessments and examine the combined impact of aid and non-aid policies on poor countries. Following an overview of policy coherence for development (PCD) monitoring and evaluation concepts and activities, the article explores evaluation options designed to account for the development impact of rich countries’ policies and to encourage greater synergy among rich countries’ policies in support of global poverty reduction. / c.19 4 июня Сивков А., 392 ^ EVALUATING CHILD AND YOUTH HOMELESSNESSThe Example of New Haven, ConnecticutEvaluating Child.pdf Following a review of approaches taken to generate accurate estimates of the scale of child and youth homelessness in America, this article documents the methods and results of a multi-pronged count of homeless children and youth in New Haven, Connecticut. The survey used in this count accessed demographics and service needs, and was administered on the streets and in a wide range of service provision settings. A total of 170 homeless families were located, comprised primarily of young, single, African American women and their children. These families included 323 children, suggesting an annual prevalence of 1,688 in this community. This work addresses the paucity of information on homeless children and youth located in small to midsized cities. / c.15 2 июня Гуркин Е., 393 ^ Evaluating Crime Prevention Scheme SuccessEvaluating Crime.pdf Research demonstrates that certain crime prevention techniques work. Accordingly, current evaluations focus on what works and where. One of the key elements in assessing crime prevention success is determining the number of crimes prevented. This allows the cost-effectiveness of schemes to be assessed and different schemes to be meaningfully compared. Evaluation studies of what works include a variety of different approaches, some more robust than others. The current article presents two methods for calculating the outcomes of crime prevention interventions. The basic principle behind both approaches involves subtracting the observed number of crimes from an estimate of the number of crimes that would have occurred had the scheme not existed. Importantly, it is acknowledged that area crime rates are influenced both by general trends and more random factors. Thus, each approach produces a range of estimates to attempt to assess the impact of more random influences. / c.20 28 мая Голодникова А., 392 ^ The Effects of Institutional Design on the Utilization of EvaluationThe Effects of Institutional Design.pdf This article presents some of the results from a study in progress, focusing on the influence of the institutional distance between evaluators and evaluees on the utilization of evaluations. The basis for the results presented here is an analysis of ten case studies from Switzerland. These cases involve evaluations that were carried out in different institutional contexts, with widely varying institutional distances between evaluators and evaluees. ‘Qualitative Comparative Analysis’ (QCA) has been used to interpret the cases, in order to allow a combination of case- and variable-centred comparisons. The analysis indicates that, under certain conditions, the institutional distance between evaluators and evaluees has no influence on the use of evaluations. In particular, formative objectives can be achieved quite independently of distance. When interpreting the results, however, one should not neglect the fact that they are solely based on a systematic evaluation of ten case studies with QCA. Generalization is not possible on this basis, nor is this the aim of the present article. On the contrary, the objective is to continue developing the debate about the influence of the institutional distance between evaluators and evaluees on the utilization of evaluations. / c.17 30 мая Ан А., 391 ^ Evaluation of Gender Mainstreaming Evaluation of Gender Mainstreaming.pdf The new policy strategy of ‘gender mainstreaming’ poses particular challenges for the evaluation of public gender-equality policies. To elaborate on this issue, a first step is to analyse the evaluation of those public genderequality policies that have formally adopted the gender mainstreaming strategy. In this article, results are summarized from a meta-evaluation of 11 evaluation processes of gender-equality plans implemented between 1995 and 1999, both at regional and national levels in Spain. This meta-evaluation focused on analysing the evaluation processes rather than the outcomes of those evaluations. First, the different types of evaluation carried out in the 11 studies are discussed. Second, some contextual factors are identified as influencing elements in the evaluations. Third, some conclusions and lessons learnt are presented, using the framework provided by the meta-evaluation criteria previously established. Finally, taking into account those lessons, a discussion of the evaluation of gender mainstreaming is presented, elaborating on the ways in which gender mainstreaming strategies and gender perspective can be evaluated and which should be used to conduct useful evaluations. / c.20 28 мая Вишневская А., 392 ^ Collaborative Action Research within Developmental EvaluationCollaborative Action Research.pdf This article investigates the use of collaborative action research in both the development and evaluation of a project designed to develop inclusive practice in Early Years and childcare settings. The purpose of the evaluation was to ascertain how practitioners understand the term ‘inclusion’, how those understandings were translated into practice and how changes in that practice might be conceptualized and carried out. The article explores the use of facilitated collaborative action research as a core element in developmental evaluation. Reflecting on other approaches such as theory of change, it considers whether the use of action research supported the critical examination and development of a project, or whether, by being so similar in design, it constricted the evaluation in terms of reliability of data, accountability and providing a framework for planning and development. / c.17 30 мая Марашов В., 392 ^ Evaluation in Multi-Actor Policy ProcessesEvaluation in Multi-Actor Policy.pdf Two main functions of evaluation are enabling accountability and collective learning. Both of these – and their combination – run into diverse complications when applied in complex multi-actor policy processes. The article explores these complications and illustrates them with examples from the field of spatial policy. In doing so, a third function of evaluation in such contexts is identified, viz. evaluation as an instrument of co-operation. Next, a number of theoretical ideas, supported by empirical research, are proposed in order to understand better when, why and how evaluation contributes to complex multi-actor policy processes. Based on these insights, some principles are elaborated for the development of constructive evaluation arrangements. The authors suggest that co-operation is a precondition for the preservation of accountability and learning functions of evaluation in multi-actor settings. / c.16 4 июня Островская А., 392 Тема №9. Основные теоретические подходы и модели оценивания How to – and How Not to – Evaluate InnovationHow to – and How Not to.pdf Many traditional evaluation methods, including most performance measurement approaches, inhibit rather than support actual innovation. This article discusses the nature of innovation, identifies limitations of traditional evaluation approaches for assessing innovation and proposes an alternative model of evaluation consistent with the nature of innovation. Most attempts at innovation, by definition, are risky and should ‘fail’ – otherwise they are using safe, rather than unknown or truly innovative approaches. A few key impacts by a minority of projects or participants may be much more meaningful than changes in mean (or average) scores. Yet the most common measure of programme impact is the mean. In contrast, this article suggests that evaluation of innovation should identify the minority of situations where real impact has occurred and the reasons for this. This is in keeping with the approach venture capitalists typically take where they expect most of their investments to ‘fail’, but to be compensated by major gains on just a few. / c.14 2 июня Чернова В. ^ Theory-based Evaluation and Types of ComplexityTheory-based Evaluation.pdf Theory-based evaluations have helped open the ‘black box’ of programmes. An account is offered of the evolution of this persuasion, through the works of Chen and Rossi, Weiss, and Pawson and Tilley. In the same way as the ‘theory of change’ approach to evaluation has tackled the complexity of integrated and comprehensive programmes at the community level, it is suggested that a theory-oriented approach based on the practice of realistic cumulation be developed for dealing with the vertical complexity of multi-level governance. / c.17 11 июня Пастухов Д., 392 ^ Economic Evaluation in the Social Welfare FieldEconomic Evaluation.pdf This article argues that economists have a potentially valuable contribution to make to evaluation in the social welfare field, provided they are willing to embrace a more flexible and eclectic approach to economic evaluation. It contrasts the perspectives of mainstream economists and other evaluators working in this field, which often appear to be at opposite ends of the spectrum. This may explain why the role of economists often seems to be marginalized – the ends don’t meet. Although different approaches to evaluation are to some extent complementary, the article argues that economists can learn from other perspectives on evaluation and should consider adapting or building on more conventional approaches to economic evaluation, where appropriate. This would help to improve the validity and generalizability of their results and would also facilitate more co-operative working with evaluators from other disciplines – helping to make ends meet. / c.17 13 июня Киселева Е.,* 391 ^ The Modus Narrandi A Methodology for Evaluating Effects of Environmental PolicyThe Modus Narrandi.pdf ‘Are we being effective?’ has become an increasingly central question in policy evaluation research. This article defends the position that this question can only be answered by employing a methodology that is adapted and can stand scrutiny in the policy field in which the evaluation occurs. In our experience the evaluation of environmental policy requires a renewed effort to develop methodologies that take into account the key characteristics of that policy area. The ‘modus narrandi’, based on causal narrative story reconstruction, is our contribution to these efforts. It is a methodology for effect evaluation that looks at effectiveness and side effects within contexts characterized by causal uncertainty. / c.22 13 июня Карпова Е., 391, ^ The Agora and the Role of Research EvaluationAgora.pdf The evaluation of research is an activity undergoing change. The traditional peer review system with its focus on scientific content and methodology has long been the backbone of research evaluation, but over the last three decades other criteria and considerations have also been integrated into the evaluation of science. This article investigates how recent societal developments – epitomized by the concept of the agora – influence research evaluations and how we perceive them. Are they still grounded in a scientific rationale, or are they more to be understood as a result of a social rationale? / c.21 4 июня Харитонова Е., 392 ^ Choosing Criteria to Judge Program Success A Values InquiryChoosing criteria.pdf What we expect from public programs determines how the programs will be judged. This article presents a systematic values inquiry used to examine how much importance citizens and stakeholders attached to specific indicators in determining the success of a public preschool program. Four groups were surveyed: teachers, administrators, parents and the public. All four groups showed significant differences with respect to the importance of 29 possible indicators. The four groups agreed on the importance of high-quality services and about the lack of importance associated with economic benefits for families and certain educational outcomes for the preschoolers. Using confirmatory factor analysis, the opinions of the groups were shown to have similar underlying structures; however, teachers made greater distinctions between the groups of potential outcomes and the public distinguished between outcomes the least. Systematic values inquiry can be an important tool in designing evaluations that will produce information that can influence the judgments of citizens and stakeholders about a program’s value. / c.21 4 июня Корниенко Е., 392 ^ Facilitating Transferable Learning through Cluster EvaluationFacilitation Transferable Learninh.pdf This article outlines a strategy for the evaluation of a development partnership within the EU ‘EQUAL’ programme. A new aspect of the EQUAL programme is the grouping of projects within thematic or geographical partnerships. This facilitates the establishment of stronger links between learning at intra-project, inter-project and programme level, in a way that was not the case in previous EU employment programmes. It provides the opportunity to address the problem of external validity (also known as transferability or generalizability), which has long been an intractable aspect of evaluation. In particular, it has been difficult to generate inter-project and policy-level learning in innovative programmes of heterogeneous projects, because learning has been strongly context-bounded. The approach of cluster evaluation has offered a methodology for bridging the gap between case-based learning and policy-level learning. The approach is suitable for thematic programmes of innovative and heterogeneous interventions in field settings with multiple objectives and diverse participants. It favours communicative learning processes among programme participants, using interactive and dialogue-focused techniques. The article reviews these issues and outlines the strategy adopted in the evaluation of a development partnership within the EU EQUAL programme in Germany. / c.15 30 мая Егоренков А., 391 ^ Information Systems for the Evaluation of the Effectiveness and Efficiency of Vocational Training ProgrammesInformation Systems.pdf The aim of this article is to discuss the requirements of an Information System (IS) for evaluating the efficiency and effectiveness of public intervention in the field of vocational training. The objective of this policy is to increase employability and reduce disparities in access to the labour market of EU-defined disadvantaged groups. The primary goal of the IS is to aid decision making of policy makers and managers. In this context, information asymmetry in the production of these services is a problem and the standard approach of the International Organization for Standardization can do little to promote quasi-market competition, which economists suggest as a solution to information asymmetry. The article recommends making a greater effort to develop an IS capable of assessing the effectiveness of training programmes and making the necessary corrections for selection bias with regard to the selection of the agencies managing the training programmes. The adoption of the standard used by current labourforce surveys is suggested. / c.17 4 июня Муравьева Ю., 392 ^ Evidence, Understanding and ComplexityEvidence, Understanding.pdf This article discusses the challenges associated with the evaluation of complex policy initiatives, focusing on one element of the national evaluation of Health Action Zones in England: Building Capacity for Collaboration. It describes the multiple dimensions of complexity that characterize HAZs and considers the limitations of the Theory of Change approach in encompassing the diverse change processes that HAZs seek to promote. Drawing on social constructionist perspectives and from both complexity theory and new institutional theory, the authors propose that evaluators need not only to be able to work with change theories generated within such complex programmes, but also to apply ‘higher level’ theory to an understanding of the way such programmes develop. They argue that the role of evaluators in ‘telling the story’ is an important part of the evaluation challenge in such contexts. / c.18 11 июня Лебедева Е.., 392 ^ The Limits of Realist EvaluationThe Limits of Realist Evaluation.pdf This article builds on an earlier experiment in applying Realist Evaluation (RE) techniques to a set of Best Value Reviews (BVRs) undertaken in a single English local authority. That experiment used a range of assumptions regarding context (C), mechanisms (M) and outcomes (O) that restricted the possible pathways in the resulting CMO causal loop. They were ‘heroic’ in nature and left largely implicit. The article subjects those assumptions to rigorous criticism. Five hypotheses are tested concerning the nature of the context within which the BVRs occurred; the potentially skewed nature of the review mechanisms chosen; the impact of process outcomes and goals-setting problems on BVR outcomes; the scope for strong linkages to be formed between context, mechanism and outcomes, such that deterministic effects ensue; and the need to define the boundaries of the evaluand (the BVR). Realist evaluations have typically focused on individual services or programmes. It is contended that RE methods need to be adapted to address cumulative impacts on policy and organizational culture that are inherently political in nature. / c.18 11 июня Салимгареев В., 392 ^ Evidence-based Policy: The Promise of ‘Realist Synthesis’Evidence-based Policy.pdf Evaluation research is tortured by time constraints. The policy cycle revolves more quickly than the research cycle, with the result that ‘real time’ evaluations often have little influence on policy making. As a result, the quest for evidence-based policy (EBP) has turned increasingly to systematic reviews of the results of previous inquiries in the relevant policy domain. However, this shifting of the temporal frame for evaluation is in itself no guarantee of success. Evidence, whether new or old, never speaks for itself. Accordingly, there is debate about the best strategy of marshalling bygone research results into the policy process. In the first of this pair of articles (published in the previous issue of Evaluation) a critical review of the existing EBP strategies was conducted. This companion article considers the merits of a new methodology for systematic reviews, namely ‘realist synthesis’. / c.18 9 июня Алехина Е., 393 ^ A Mutual Catalytic Model of Formative EvaluationA Mutual Catalytic Model.pdf This article describes the ‘mutual catalytic model’ of programme design and formative evaluation. The model presumes two conditions under which it can be useful. Firstly, those responsible for a programme at the point of service delivery should participate equally in a programme’s design and in its evaluation. This collaboration will: 1) increase the clarity of programme objectives and the level of consensus regarding them; 2) improve the evaluation design and process by connecting the evaluation with the realities at the point of service delivery; and 3) increase the likelihood that staff will use the results of the evaluation to improve programme design. Secondly, evaluators are required relatively frequently to facilitate the review of programme objectives and procedures, since reviews of this kind rarely take place without an extra stimulus and sometimes need a practical hand to guide them. In the mutual catalytic model of formative evaluation, local practitioners help to specify programme goals and improve evaluation, while evaluators help to elaborate programme goals and evaluation needs. This model is exemplified using experiences with homeless-child care centres in Seville in Spain. / c.17 13 июня Якушкина Н., 391 ^ A Framework for Evaluating Environmental Policy InstrumentsA Framework for Evaluating.pdf The development of environmental policy evaluation has been slow, compared with many other realms, and many practices are still not well standardized. Environmental problems have some key features; for example they often have long time frames, are complex and concern geographically remote regions. Scientific knowledge is important in environmental policy but such knowledge is characterized by huge uncertainties. The key characteristics of both the problems and our knowledge of them should affect the evaluation of the policy instruments used to address them. An evaluation of Finnish wastewater permits is used as an example throughout the article. If evaluation of environmental policy is undertaken without due consideration of the specifics involved there is a great risk of identifying only minor impacts and low effectiveness. Properly conducted evaluations, however, can help us to make better decisions on both old and new environmental policy instruments. / c.20 2 июня Семенушкин И., 393 ^ Choosing Evaluation ModelsChoosing Evaluation Models.pdf A variety of different evaluation models are presented in the evaluation literature. These mostly fall into the following categories: results models, process models, system models, economic models, actor models, and programme theory models. This raises the question: ‘how can evaluation sponsors and evaluators decide how to design an evaluation with so many models to choose from?’ In this article, several – mutually incompatible – recommendations are discussed. Design should be determined by the purpose of the evaluation, the object of evaluation or the problem to be solved by the evaluated programme or agency. The recommendations are based on different rationales: goals–means, context-based values and programme theory. Furthermore, in practice other logics may influence the evaluation design processes. Four hypotheses concerning such logics are proposed: negotiation, appropriateness, ‘routine’ and projection of competence. / c.14 30 мая Ахаев Д., 391 ^ A Theoretical Framework and Grid for Analysis of Programme-evaluation PracticesA Theoretical Framework.pdf This article proposes a theoretical framework and a grid to unify models and to describe programme evaluation. It identifies three components in the evaluation process: the information-collecting process; the value-judgement product; and the use of the latter in decision making. It defines three differentiation axes (or dimensions): the social enunciation, the external or internal position of actors, and the formal or informal character of the evaluation components. The relevance for the practitioner and the researcher is discussed and briefly illustrated with two examples from the health promotion sector: a project selection process and a retrospective study of programme-evaluation practices. / c.18 9 июня Черкасова Ю., 393 ^ The European Structural Funds and Sustainable Development A Methodology and Indicator Framework for EvaluationThe European Structural Funds.pdf Sustainable development (SD) is now a broad policy objective in The European Union (EU). This article explores how the contribution to SD of the EU Structural Funds (SFs) mi