ACTION EVALUATION IN THE THEORY AND PRACTICE OF CONFLICT RESOLUTION¹

Marc Howard Ross 

Abstract 

        Questions of evaluation are important to conveners, participants and funders of conflict resolution initiatives. Yet good evaluation is tied to a number of complicated questions concerning what constitutes success and failure in projects that may be multi-dimensional or only part of an effort to settle a larger conflict. Rothman has offered Action Evaluation as a methodology that seeks to incorporate goal setting and evaluation into project designs. He argues that this will improve a project by monitoring the changing nature of goals through the life of a conflict resolution intervention, and action evaluation’s self-conscious attention to goal setting offers a mechanism for developing and committing an intervention to specific internal and external standards of evaluation. This article examines Action Evaluation as a theory of practice, considering its conceptual strengths and examining specific issues of its implementation. 

Introduction

        How are conveners, participants and funders to decide if a conflict resolution initiative has been successful? The question is not easy to answer, especially in situations where a project suggests that its impact will be indirect and not necessarily visible in the short run. To date various inadequate solutions to this question of evaluation of conflict resolution initiatives have been proposed.² Action evaluation offers a different approach to evaluation as it seeks to incorporate goal setting and evaluation into project designs, to recognize the changing nature of goals through the life of an intervention and to use a self-conscious attention to goals as a mechanism for developing and committing an intervention to both internal and external standards of evaluation (Rothman, 1997).³

        My own interest in action evaluation comes from a concern with understanding the diverse goals of conflict resolution interventions in ethnic conflict (Ross and Rothman, 1999), and in puzzling over the question of how social science theory and methods can be more closely tied to practice. My approach to examining theories of practice in conflict resolution is to take seriously practitioners’ underlying, often unstated, assumptions about conflict and steps they take to deal with it. These beliefs, I have argued, are often central to understanding why the parties to a conflict act as they do, and why specific conflict management interventions are organized as they are (Ross, forthcoming). Making the core assumptions of practitioners explicit allows us to spell out their theories of practice to better understand actors’ motivation, to evaluate the extent to which their core beliefs are consistent with theory and evidence and to refine both theory and practice to the benefit of each.

        I use this approach to examine the theory and practice of action evaluation asking what its core assumptions are and trying to make explicit how it seeks to achieve the specific effects it wants. Rothman proposes action evaluation as a method for improving practice and potentially contributing to a clearer understanding of the theory underlying conflict resolution as well. “This methodology is intended to help project organizers, facilitators, participants, and funders interactively define their shared goals, as their project evolves and effectively monitor and assess them” (Rothman,1999b 2).

        In this article, I first discuss the concept of theory of practice; next, I examine action evaluation; then, I consider how action evaluation has been used in specific conflict resolution projects, and lastly, I suggest areas where the theory of action evaluation requires further development, additional theoretical specification, and empirical analysis. 

Theories of Practice 

        All practice is grounded in beliefs about the nature of social and psychological reality. These beliefs, which help us understand why and how practitioners’ activities produce their intended effects, are often more implicit than explicit. Making them explicit permits us to identify the core assumptions of specific theories of practice, to articulate indicators that could help us evaluate if given theories are correct, and to revise practice when core assumptions are imprecise or unwarranted.

        Any theory of the practice makes important assumptions about the following: the nature of conflict with an emphasis on the specific domains or “problem areas” to which it pays particular attention, the specific effects good practice is expected to have on participants in interventions, the possible impact a practice can have on the course of a conflict beyond those it has directly on the participants in an intervention, images of what a successful settlement of conflict looks like, and hypotheses about the mechanisms by which the project achieves its impact (Ross, forthcoming). Elsewhere I have examined six different theories of practice in ethnic conflict resolution, community relations, principled negotiation, human needs, psychoanalytically rooted identity, intercultural miscommunications, and conflict transformation, in an effort to spell out the very different activities and contrasting ideas about success which are consistent with each of these theories (Ross, forthcoming). That analysis showed that while alternative theories rarely directly contradict each other, they do emphasize quite different processes and sequences of activities. For example, where community relations focuses on local institution building and empowerment, human needs theory stresses the identification of the parties’ non-negotiable, underlying needs and consideration of how the needs of each party are often not incompatible with those of the others.

        Central to my argument is that underlying all theories of practices are judgments about what success and failure in conflict resolution entails. What does it mean to settle, resolve, or manage an ethnic conflict successfully?⁴ An examination of different theories suggests significant variation in the criteria of success which each does (or could) articulate and emphasize. Equally important, particular approaches to conflict resolution differ in how they envision what Kelman (1995) calls the “transfer process,” the linkage between how the effects of conflict resolution are extended from those relatively small number of people who participate directly in conflict resolution activities and changes in the larger conflict between ethnic communities.

        Different notions of what is success follow from different theories of practice. For example, although each of the six theories of practice I mention above can be said to share the goal of resolving ethnic conflict, there are significant differences between them in what exactly this means (Ross, forthcoming). For example, community relations work seeks to improve communication and intergroup understanding, promote tolerant acceptance of diversity, and encourage building structures that safeguard the rights of all. Principled negotiation tries to bring about positive sum (win-win) agreements between the parties. Human needs theory emphasizes that the recognition of how each party in a conflict has some similar needs and is a prerequisite to joint action. Psychoanalytically informed identity theory tries to build analytic empathy⁵  between the parties, to encourage a sense that agreement between the parties is possible and to lower the parties fears so they are more able to explore alternatives to continuing confrontation. Intercultural (mis)communication theory’s goal is to enhance effective communication by increasing the parties’ knowledge of each other and by weakening negative stereotypes. Conflict transformation theory attempts to change relationships among the parties through moral empowerment, justice, forgiveness, reconciliation, and recognition.

        The capacity to process information from one’s environment as a basis for choosing specific actions is a central feature of human behavior. To process new information from the environment as a first step towards undertaking action requires the existence of some sort of model of the world, what many social psychologists and cultural anthropologists call a schema, which is used to interpret what the new information means and its consequences for action (D’Andrade, 1992). The schemas social psychologists describe contain assumptions about how the world one lives in works, about the motives of different social actors, and about the consequences of action on others. All social actors possess such schemas or theories about the world. Schemas differ from each other, however, in how explicit and elaborate they are.⁶

        At the core of theories of practice are the principles that guide action. Making these principles explicit is important because it allows the different stakeholders in a project to discuss them and to consider how they are tied to a project’s goals. Action evaluation recognizes that self-consciousness about a project’s core assumptions and about stakeholders goals is not automatic. Rather, it argues that specific procedures are often needed to increase self-awareness, active reflection and choice making to guide projects as they evolve and as they try to decide when, and how, they have been successful. As a result action evaluation contains its own theory of practice at the core of which is integrating careful goal articulation and monitoring into practice more generally, will facilitate project design, promote effective evaluation, and improve the validity and reliability of ethnic conflict resolution efforts. 

Action Evaluation

        Rothman (1997) presents action evaluation as a method for integrating evaluation into the practice of conflict resolution training and interventions. His starting point is the belief that current conflict resolution efforts are poorly served by standard pre and post training evaluation in which participants are asked the extent to which broad general goals articulated at the outset of a project, such as an intensive training, have been achieved, or where conveners develop a post-hoc imprecise design to decide what is the impact of their project. Instead, action evaluation encourages an active and continual focus on goal definition and achievement throughout an intervention. Through self-conscious engagement with project goals and their evolution, participants can become much more explicit about what, why, and how they are seeking particular goals and in the process they become far more committed to their achievement. Furthermore, Rothman (1999b: 2) hypothesizes that such self-conscious engagement in also likely to raise the chance for success.

        Many conflict resolution interventions, Rothman suggests, are motivated by the moral importance of the conflicts they seek to resolve rather than explicit project goals which drive the specific daily activities of the interveners. As a result, it is often difficult to know the extent to which a project’s activities affected either the participants with whom they work or the larger conflict in which the intervention is embedded. A parallel problem is that too many projects engage in activities that are not clearly connected to a project’s goals, often because the goals are formulated too vaguely for projects to link activities and goals effectively.

        Action evaluation incorporates goal setting, monitoring and evaluation into a conflict resolution initiative rather than seeing these as distinct activities to be conducted independently and at different points in time. It seeks to make explicit the goals and motivations of all stakeholders, to analyze how these evolve over time, and to encourage the stakeholders to use the goals which have been identified as a step towards identifying explicit, contextually defined, criteria of success by which a project might be judged.

        Action evaluation is a goal driven process in several senses. First, it seeks to make explicit the wide range of goals that inform an intervention. It does this by asking the relevant actors in an intervention to identify their goals, to say why they care about them (what motivations are driving them), and to identify how they think the goals can be most effectively met. Goal statements are collected at various stages from a project’s organizers and the participants in the intervention. In addition Rothman (personal communication) seeks to engage funders as active and explicit partners in the goal setting process to help develop more realistic and partnering attitudes among funders and to shape future funding policy.

        Second, through the work of a project member, the action evaluator,, and an individual particularly charged with the responsibility for collecting and analyzing the project’s goals, ,the goals are then summarized and presented back to the stakeholders in a project. This is done in several stages and at several levels of aggregation to establish a project’s baseline goals, to identify the shared, divergent and unique goals within and between the conveners, participants and donors, and then to map the goal evolution. By making the participants self-conscious about their project’s goals, action evaluation seeks to promote a reflection about, and shared commitment to, the project itself.

        Third, action evaluation seeks to use the process of tracking and monitoring goals as a way toward developing contextualized standards (or criteria) of success, which can be employed for internal and external evaluation. The internal standards are needed if a project is to be self-correcting as it reacts to both changes in the conditions of a conflict, and as it learns which of its goals (i.e. those which all stakeholding groups have set) have or have not been successfully achieved. External standards are those that outside evaluators and others can use to determine the extent to which a project has established and met meaningful goals in terms of the larger conflict in which it is embedded.

        Core assumptions of action evaluation. Underlying action evaluation are several crucial assumptions that are consistent with a great deal of social science theory and evidence which offer strong support for the method.

        The participation hypothesis. A basic and well-supported proposition underlying action evaluation is the idea that people will be more committed to goals that they articulate and establish themselves. Dubbed the participation hypothesis, Verba (1961: 206-43) and many others find a great deal of evidence for the proposition that active involvement in a process builds commitment. The participation hypothesis suggests initial commitment can first be built by eliciting goals from participants and that additional commitment occurs when participants are asked to join together to reflect upon project goals. One reason is because people become invested psychologically when they spend time on an activity. Another is that participation builds a new social identity that is sustained, at least in part, by working toward common goals. Lederach’s (1995; 1997) concept of elicitive conflict resolution has participants define a situation and design their own contextually relevant action program; it clearly builds on the participation hypothesis’ emphasis on participants’ motivations and commitments.

        Action evaluation’s impact, and the participation mechanism, also finds support in the Hawthorn effect; that is, the fact that participants at all levels are asked questions and involved in the process of program design builds support for the program and increases commitment to its goals. While some view this effect as an example of the problems of doing field research, the action researcher sees this finding as an opportunity to direct an outcome in a favorable direction (Argyris et al, 1985). From this point of view, it is not the specific goals which participants identify which becomes crucial in the process as much as the involvement in the process that increases their engagement in the process and their desire to achieve successful outcomes.

        As part of action evaluation’s attention to participation in goal setting and evaluation is its implicit attention to strengthening cognitive and affective links among participants through their involvement in goal identification and achievement. The method, as Rothman has developed it to date, stresses the Action Evaluator as the core person collecting and analyzing the goals of different participants. However,  it should also be pointed out that action evaluation also promotes discussion and negotiation about goals among conveners, participants, and funders and pays particular attention to having each group consider similarities and differences among their members with its attention to shared, unique and opposing goals as part of their self-reflection and mutual engagement.

        Goal setting as an iterative, incremental process. Action evaluation is rooted in the premise that goal setting and evaluation are iterative processes that reflects both participants’ changing concerns, understandings, and the shifting contexts in which conflicts are situated. As a result, action evaluation explicitly rejects the notion that it is desirable for initiatives to fully articulate project goals at the outset and to fail to modify them over time. Rather, it is based on the belief that incremental (and sometime large scale) changes in goals should be incorporated into project designs.

        Effective feedback, of course, is crucial to any interactive process of goal modification and action evaluation provides at least two different kinds of feedback to participants. One asks the Action Evaluator to summarize and analyze the participants’, conveners’, and funders’ goals which are presented for discussion to each group at various points in time. In addition, the method also asks the Action Evaluator to track goals, changes, and continuities over time and to feed these back to participants as well. The objective here is to make participants more self-consciously aware of how their thinking and that of other participants has shifted as a mechanism for building commitment to the achievement of a project’s  evolved goals.

        The emphasis on iterative, incremental processes in action evaluation builds upon the analysis of effective organizational decision-making which Lindbloom (1959) and March and Simon (1958) provided more than a generation ago. They argued that problem identification and the development of solutions occurs in a context of imperfect information and changing priorities and understandings. In such settings, good decisions, what they call satisficing rather than optimizing ones, are those which result from continuing responses to changes and feedback, not large scale, one-time actions. Certainly March and Simon’s ideas about limited rationality apply to most conflict situations, and therefore action evaluations iterative, incremental approach is likely to do better than broad top-down procedures which are not subject to regular self-monitoring feedback and adjustment.

        The social construction of goals. Action evaluation emphasizes that project goals need to be both specific and contextually relevant. As a result, an implicit objective of the method is to make participants seriously reflect on and discuss their goals, so they will be less likely to accept vague, general goals such as bringing peace to a long-time troubled region. Instead a central part of the action evaluator’s task is to help the different stakeholders articulate more specific goals and to be aware of their reasons for holding them. The process of self-conscious reflection seeks to get people to articulate detailed, meaningful objectives through an iterative process involving goal setting, discussion, and action across stakeholding groups.

        Because action evaluation obliges participants to discuss their goals, and the motivations underlying them, and to suggest how they think their the goals can be most effectively met, action evaluation pushes all stakeholders to consider the relationship between their goals and a project’s capacity and its specific activities. As a result, the formation and explicit articulation of objectives is understood to occur in a social context and is promoted through the active engagement of the Action Evaluator. This process recognizes not only the social nature of goal construction, but also that it is a process that can be nurtured and encouraged when interventions wish to pay attention to it.

        The social nature of goals is linked to action evaluation’s emphasis on an active process of goal setting and analysis. While different groups of stakeholders may not always be comfortable articulating and examining their goals, action evaluation implicitly suggests that the social dynamic engendered through the process creates its own social context that can foster group identification and commitment. What is left somewhat ambiguous in Rothman’s formulation is the extent to which the emerging social ties among stakeholders are simply functional, working relations and the degree to which they are to be affective as well.

        Theory and practice are interrelated, not separate, phenomena. Rothman cites John Maynard Keynes’ famous remark that there is nothing so practical as a good theory to emphasize the importance of strongly linking theory and practice. Action evaluation is founded on the belief that reflexive practice must take theory seriously and that good theory must find strong support in practice. The linkage between the two is actively sought in action evaluation by forcing practitioners to articulate their core assumptions while recognizing that this is not always easy to do. In fact, a key role of the Action Evaluator is to help those involved in a project to do this where they are not fully comfortable with the process, and to translate the specific, operational statements of practitioners into more theoretical terms. 

The Practice Of Action Evaluation

        Much of Rothman’s inspiration for action evaluation comes from the long tradition of action research, which he traces from Kurt Lewin to Chris Argyris. Rothman argues just as good theory and practice each improve the other, well-done evaluation is necessary to improve practice. To link practice and evaluation, he has tried to develop a strategy which forces conflict resolution interventions to pay more explicit attention to the analysis of project goals. As discussed in the previous section, Rothman’s approach assumes that a self-conscious focus on goals will (1) help clarify them for stakeholders, (2) move stakeholders to a consensus on appropriate, contextually defined goals for an intervention, and (3) assist in the definition of standards to evaluate the extent to which a project has or has not been successful.

        Each project’s Action Evaluator plays a key role in collecting and analyzing goals, but the process also requires that other members of a project see the value in what can be a somewhat tedious and time-consuming process. In addition, his approach requires that participants are willing and able to articulate their goals and trust the process will meet their needs and interests. But this doesn’t occur all at once.  

            [Rather] this is an ideal towards which action-evaluation strives and if it is successful develops over time. The Action Evaluator is the first repository of this confidence that is then widened to include conveners, participants, and perhaps, funders. A long and probably never completely successful process of transfer is clearly better than non-efforts made for inclusion and buy-in. At least all voices are heard” (Rothman, personal communication). 

        Perhaps the best way to understand how action evaluation works is to examine it in the context of a few of the more than a dozen projects in which Rothman and his associates have used it to date. While I will describe action evaluation in several different interventions, my discussion of its use is limited by the fact that in no case yet did an intervention use action evaluation for a long enough period to track important changes in goals over time; nor has one yet proceeded to the point where the process produced clear standards for evaluating the project’s success as Rothman hopes the method will do when it is carried out over a longer period of time.

        Communication/Decisions/Results (CDR) Associates’ work with the Stara Zagora Multi-Ethnic Commission in Bulgaria illustrates some important dynamics of action evaluation and how its use is tied to a specific context. CDR’s project in Bulgaria sought to build cross-ethnic cooperation between the Bulgarian majority and several minority groups including the Roma and Turks. As part of its work, through a partnership with the Foundation on Negotiation and Conflict Resolution in Sofia and the Open Education Centre, CDR helped establish multi-ethnic commissions that seek to address local, and especially, minority problems in several Bulgarian towns. Action evaluation began with interviews with four stakeholder groups: sponsors, supervisors, conveners, and participants (Ghais, n.d.: 2). Ghais’ analysis of the data from these interviews showed important differences in emphasis within and between the stakeholder groups.

        The participants saw the Commission as a potentially important way to help the underprivileged minorities in Bulgaria. Ghais points out the participants’ commitment to joint problem solving and a sense of optimism among them that the Commission can make a real difference in the lives of people in the community. The interviews with participants made a number of explicit references to minorities’ social and economic problems that they wanted to see addressed, perhaps in coordination with local government and non-governmental organizations (NGO’s).

        The conveners, while also expressing an interest in helping needy minorities, placed more emphasis on improving interethnic relations though increasing tolerance, conflict resolution, social integration, and even the use of the Commission as a model for interethnic cooperation in Bulgaria. The supervisors while sharing the goals of enhancing intergroup understanding and the development of effective models, also wanted to enhance their own experiences and knowledge about the cultures of Bulgaria’s minorities and their own conflict resolution skills. Finally, the sponsors emphasized the Commission project as part of Bulgaria’s transition to democracy and as a way of creating a culture of democracy and dialogue.

        In analyzing differences in specific goals across groups, Ghais (n.d.: 7) points out: 

            The four groups fall along a spectrum in terms of their goals for the Stara Zagora Commission. At one end of the spectrum, the participants seem most concerned with helping minorities through charitable work: helping them find jobs, improving education and health care. (The conveners share these goals but also hold others.) At the other end of the spectrum the sponsors see the project as part of building a culture of dialogue and democracy. This spectrum of goals can also be see as going from tangible, results-oriented goals (providing relief for the problems of the poor) to more intangible, esoteric goals (instilling a culture of dialogue).  

        In her role as Action Evaluator, Ghais found that in response to the question of how they might accomplish their goals, there was agreement across all four groups concerning their desire to institutionalize and strengthen the Commission and to bring people from different ethnic groups together. Furthermore, none of the goals of any of the groups are incompatible with those others identified. At the same time, there are clearly differences in priorities and she concludes that, “Which of these activities are given priority depends on which understanding of the nature of the Commission prevails. If the goal is to bring about intergroup harmony in Bulgaria, a particular goal such as helping children stay in school is less important than the intergroup collaboration in achieving any goal” (Ghais, n.d.: 7).

        In this project, the Action Evaluator prepared an analysis of the different stakeholders goals which the supervisors and conveners then reflected on with the aim of reaching agreement on the direction of the Stara Zagora Commission’s activities and which was then to be discussed with the participants. The aim of such analysis and discussion is to raise awareness concerning differences in emphasis as well as areas of agreement while moving stakeholders toward consensus and clearer shared understanding of where a project should be headed. The practice of action evaluation views baseline data such as these as important because they help stakeholders understand both their own and others’ goals after the Action Evaluator (and possibly others on the project) analyze the data and present it back to the stakeholders in a useful form. Exactly what form goals should be presented to stakeholders will vary across projects and cultural contexts. They might be presented in a written document or orally; it can begin with separate meetings for each stakeholder group but can easily move toward sessions with more than one and joint exploration of both their similarities and differences as well as the future direction of a project.

        Examining several other projects it is clear that the baseline data the Action Evaluator collected are important in revealing very different emphases among stakeholders, and the systematic analysis of stakeholder goals forced the project to consider how they might be incorporated into the project’s work rather than simply smoothed over. For example, in the Zichron Forum project in Israel while many of the founders were intellectuals most interested in fostering a dialogue between religious and secular Jews, the primary concerns of many of the participants were with social welfare issues. As the differences between the groups surfaced, there was a great deal of rancor as each tried to assert the priority of its goals. Finally, the action evaluation process led the participants to recognize that the different goals were not necessarily contradictory and to recognize it was possible to attend to both sets of goals. Similarly, the Action Evaluation data in the CIC Project in Yellow Springs Ohio, a forum to address issues of the town’s development, revealed a sharp split between people favoring social and cultural improvement of the community and those emphasizing economic development. The process which identified differences at first made participants uncomfortable because of the different directions each orientation would move the project However, when the Action Evaluation project also encouraged the stakeholders to work with, and address, their differences, rather than pretending they didn’t exist, the participants were more comfortable.

        To date no project has yet employed action evaluation from initiation to conclusion of its work. The spirit of action evaluation, however, encourages us to reflect on the practice as it develops, rather than waiting for the completion of one or more applications before reacting to it. Clearly, a core strength of action evaluation is its capacity to build empowerment through the encouragement of stakeholder awareness of their own and others’ goals and motives. Used effectively, we might expect action evaluation to help projects evolve and persevere where many might otherwise expect them the end. At the same time, at the early stage of the development of this practice, there are still issues needing further attention, a question to which I now turn.  

Issues Needing Further Attention in Action Evaluation

        There are a number of theoretical and practical issues that require further attention as action evaluation develops. The seven issues I raise ask both how interveners can integrate action evaluation into their work and suggest avenues for additional research and theory development that would demonstrate why and how action evaluation can improve conflict resolution interventions.

        The role of the Action Evaluator. In several of the projects in which Rothman has piloted, the Action Evaluator has felt frustrated and was not certain that the conveners or participants were committed to the method and its procedures. In some cases, this was because action evaluation takes time, something that is not always readily available. In other cases there were concerns about confidentiality and fears that direct questions about goals might produce more problems than returns. Another issue is that sometimes the designated action evaluator was not a full member of the project team and their work was too easily seen as being at cross-purposes with the initiative. Some project conveners in some cultural and political contexts are uncomfortable with the direct questions action evaluation poses to participants and believe they may even anger and alienate some participants. More generally the concern of some is that action evaluation’s step-by-step process is inconsistent with how many non-western cultures approach problem solving.

        All of this means the role of the Action Evaluator needs to be more carefully thought out and perhaps Rothman needs to consider a range of ways in which the role can be filled. At the same time, whatever decision is reached about the Action Evaluator’s role in any project, there needs to be a widespread project commitment to action evaluation for it to be successful. Without support conveners or participants who want to undermine the process can easily do so.

        Making goals explicit and monitoring changing goals increases the likelihood they will be achieved. At the core of action evaluation is the hypothesis that making goals explicit and monitoring changes in stakeholder goals increases the chances that the goals will be met and that an intervention will be successful. While Rothman is probably right that increasingly self-awareness of goals is linked to commitment to their achievement, action evaluation needs to be more explicit about why this is the case, to identify situations in which this proposition is particularly like to hold and to consider others in which it is likely to be more problematic.

        There are several different possible underlying dynamics at work here. As is suggested above, each could have somewhat different implications for practice. First, it may be that focusing on stakeholder goals is an effective mechanism to increase commitment to a project, as participants feel empowered because they are asked about their priorities. Second, it may be the case that clarification of goals and their prioritization makes people more focused in their project activities and this increases their effectiveness as participants’ data and input are used to design (and redesign) initiatives. Third, it may be that identification of, and attention to, specific goals heightens stakeholder’s motivation. Fourth, it is plausible to suggest that eliciting goals and discussing them heightens the social and emotional connections within and between different stakeholders that provides a cadre of persons prepared to work for the resolution of the larger conflict. While each of these mechanisms are plausible and not necessarily at odds with each other, collecting evidence on the extent to which each is operating and the strength of their effects is necessary to support claims about how and why action evaluation is effective.

         Action evaluation may be far more appropriate in certain kinds of conflicts than others. It is worth considering the conditions under which the conflict resolution mechanisms at the core of action evaluation are most likely to comes into play and where goal identification and analysis are most likely to move a conflict closer to resolution. (The reverse is to consider situations when they are likely to be particularly ineffective.) Asking this question reminds us of possible limits to conflict resolution more generally, and how in some intransigent conflicts explicit attention to disagreement about goals can sometimes harden differences among the parties. As a result, there may be differences across conflicts in the extent to which action evaluation can be effective, and identifying some of action evaluation’s limits may be particularly useful as interveners decide whether it is appropriate in the conflict on which they are working.⁷ It is easy, for example, to imagine bitter, intractable conflicts where the parties are not yet ready to share their goals with opponents in any kind of frank and open process which action evaluation requires.

        Participants in an intervention can effectively develop criteria of success to evaluate the extent to which the larger conflict is or is not moving toward resolution. One of the most interesting ideas action evaluation develops is the process of goal-setting among stakeholders in an intervention can lead to the development of meaningful standards for evaluation, contextually for individual projects and then across projects. What is not spelled out however is how it occurs. One potential problem is that not all people involved in an intervention necessarily are comfortable thinking in operational terms and can easily identify specific indicators of success. Getting people to be sufficiently operational may not be so easy in some contexts, and many conveners will probably feel tension because of the time and energy it requires and the stress it produces on some participants.⁸

        In addition, while particular stakeholders may develop clear operational goals, there is not necessarily any assurance that there will be agreement across stakeholders or between different groups of participants about what the goals should be or how they might be measured. While Rothman suggests that negotiations among the stakeholders are necessary to achieve agreement on goals, it may be the case that problems are an indicator of the larger conflict and not something that stakeholders can easily negotiate. When presented with such a dilemma, conveners will have to make decisions about how much time and effort to devote to this process and when avoidance of explicit differences among participants is the best short-run strategy. There may then be strong differences between how conveners think it is best to proceed and the what the action evaluation process asks them to do.

        Forcing participants to establish a common set of goals may result in a tendency to accept the least common denominator, related to only those few goals that are not controversial and relatively easy to achieve. One possible outcome to a difficult situation is that participants will only agree on those few, general goals that are either not problematic and/or relatively easy to achieve. While this meets action evaluation’s demand that specific goals be articulated, it may undermine the overall value of an intervention. In addition, a too-narrow demand for agreement on specific goals may create real tension between the Action Evaluator and other members of the project in ways that turn many participants against the action evaluation process.

        The problem here is that action evaluation ,the process of articulating and monitoring goals, can have a great deal of tension associated with it, and there needs to be more explicit attention paid to how to deal with situations where action evaluation’s procedures are an important source of stress. The Action Evaluator and other conveners may come to believe that alternative, more indirect, approaches to goal articulation and monitoring are needed, and yet action evaluation as Rothman spells it out does not make clear how this might be achieved.⁹

        Action evaluation generates many goal statements, yet it not yet clear what is the best way in which these are to be analyzed and how their analysis best ties into conflict resolution goals. Collecting goal statements from different stakeholders in an initiative is one thing; deciding how to makes sense of and use these statements is another. Before developing action evaluation, Rothman (1999a) reports an intervention in the Cyprus conflict in which he sorted over two hundred goal statements into ten groups on the basis of their similarity in content. He expresses the hope that such categorization and sharing the groups of goals with participants will help them see connections among goals and can help interveners understand general goals for the field. However, it is not yet clear what the connection is between goal classification and action.

        Action evaluation needs to explore additional ways to analyze and use the goals stakeholders identify. One avenue to develop involves goal prioritization. At the simplest level, this is about rankings and distinguishing among goals that are ranked highly and those that are not. A further step might be to identify particular goals stakeholders see as critical or essential from their perspective and to emphasize their importance in any analysis or group discussion of goals, especially in situations where they may be at odds with other’s high priority goals. At present degrees of agreement across goals are calculated, but a measure of intensity could also be developed to get at prioritization.

        Another possible dimension for goal analysis would distinguish among process goals involving the participants in the intervention and substantive outcome goals. The latter could be divided between outcome goals which primarily involve the intervention participants and those which concern the wider conflict; an example of the first might be that participants in a workshop develop a keener appreciation of the other side’s perception of what is at stake in the conflict, while the latter could be that confrontations and violent incidents between groups in a town diminish over the next year.

        Action evaluation is ultimately a form of third party intervention and as such must be evaluated as other forms of intervention are.¹⁰ Emphasizing goal articulation, their explicit recondition and efforts to build consensus around project goals, their prioritization, and criteria of success, does not obviate the need to ask whether this is the ‘best way’ of intervening in any given conflict. Action evaluation emphasizes goal articulation and the definition of success, but in fact has little to say about how success should be measured or how measures of success it develops might differ from those that other methodologies generate. In fact, it is important to recognize that the internal generation of goals can, at times, be self-serving and collusion among different active participants may result in avoiding difficult problems. 

Conclusion

        This paper has attempted to describe the key elements and core assumptions of the practice of action evaluation. I have argued the approach makes four crucial assumptions consistent with a great deal of social science theory and evidence: the impact of participation on attitudes and behaviors, goal setting as an iterative and incremental process, the social construction of goals, and the interrelationship between theory and practice. These assumptions are important in helping us understand why Rothman advocates action evaluation as a way of building stakeholder awareness of, and commitment to, goals in conflict resolution projects.

        Action evaluation is a strategy for making stakeholders in conflict resolution projects pay explicit attention to their own and others’ goals and motives. The underlying hypothesis is that self-reflection assists stakeholders in clarifying what it is they want a project to accomplish, showing how their own goals fit with those of other participants, and discovering new, evolving goals as projects proceed. Through such a process people come to prioritize their own goals and build commitment to others in a project. Finally, the explicit nature of project goals can come to serve as the basis for standards for evaluation that allow both project participants and those outside a project to decide ways in which it has or has not been successful.

        Evaluation from this perspective is far different than the alienating process in which outsiders use externally derived standards to decide when a project is a success or failure. Instead, it is far more interactive and gives project participants ownership of (and responsibility for) the criteria of success by which a project will be judged.

        Agreement among stakeholders, however, is not something that one can reasonably expect to be achieved quickly or easily. In fact the importance action evaluation places on the role of the Action Evaluator recognizes that questions of what constitutes success and possible areas of disagreement are often matters which stakeholders seeks to avoid, as they can be sources of discomfort and rancor. The Action Evaluator wants to use differences in goals and priorities among goals to foster reflection, choice making, and exploration of new alternatives. Through such a process conflict resolution projects can be clearer about what is most important and stakeholders can develop criteria by which success and failure can be meaningfully evaluated.

Notes

¹ Support for this research was generously provided by Jay Rothman’s Project on Action Evaluation with support from the Pew Charitable Trusts.

² My focus here is not on the various evaluation instruments projects have designed or on the different methods of outside evaluation that have been used. It is worth noting however that often projects design instruments which are viewed by funders and others as self-serving when they do little more than garner participants’ attitudes towards the intervention, and frequently conveners feel that external evaluation is unfair when it imposes evaluation standards on a project which are not those the project thought it was working towards.

³ Internal standards involve direct effects on project participants, while external standards are those that concern a project's impact on the larger conflict in which it is embedded. For example, a project might have the goal of increasing contact and discussion between members of two hostile ethnic communities. This internal goal is distinct from the external goal of lowering tension between the two communities from which the participants in the project come (Ross and Rothman, 1999).

⁴ Here I do not consider differences between conflict resolution, conflict management, and conflict settlement, although I recognize that different theorists and practitioners often strongly prefer to use one or another.

⁵ By analytic empathy I mean the capacity to understand, but not necessarily sympathize with, an opponent's position.

⁶ Anthropologists interested in schemas distinguish between folk and social-scientific theories in two ways (D’Andrade, 1992). Folk theories are those of local actors in particular situations and are more likely (but not necessarily) to be implicit. In contrast, social-scientific theories are the explicit, more general, theories social scientists use to explain social action in more than one context. Where by definition folk theories utilize actors’ own concepts and frames of reference, social-scientific theories are more likely to employ concepts that many actors do not recognize or use themselves. This is not surprising since social scientists attempt to develop general theories that can be used across contexts and folk theories are contextually specific. However what both folk and social-scientific theories have in common is that each are generalizations about the world and are efforts to make sense of it. The distinction between the two kinds of theories is one of degree, not kind, and my concern here is to emphasize that both offer guidance for action and that underlying conflict resolution practice are important assumptions about why and how certain actions matter. While it is certainly the case that many will argue that good practice needs to consider both folk and social science theory, here I draw attention to epistemological and methodological differences between them that often make it difficult to integrate the two. For a discussion of the same two perspectives in conflict resolution, see Lederach's (1995: 37-72) discussion of prescriptive and elicitive approaches to training. Despite differences in terminology and explicitness both folk and social scientific theories are generalizations about the world relevant for understanding action. Both folk and social scientific theories can be articulated and help us understand what people think can or cannot be done to manage them constructively. However, we should also recognize that while both inform action, because they are very different forms of knowledge, they can affect behavior in very different ways.

⁷  Rothman (personal communication) adds while this last point emphasizes action evaluation’s role in highlighting and clarifying differences, much of his initial data show the presence of different (but not necessarily) incompatible goals, and more important, active goal articulation can move a process towards consensus and common ground where one did not exist before.

⁸ Sometimes, for example, explicit goal setting may overly emphasize cognitive processes when the core of a project is to produce changes at the affective level.

⁹ Rothman reports that different methods for gathering and evaluating goals are currently being compiled in a handbook under preparation. Furthermore, he notes that the collection procedures range from formal interviews to more ethnographic methods in which participant goals are inferred from statements in meetings and their behaviors.

¹⁰ I am grateful to an anonymous reviewer of this article for encouraging me to emphasize this important point. 

References

Argyris, Chris Robert Putnam, and Diana McLain Smith. 1985. Action Science: Concepts, Methods, and Skills for Research and Intervention. San Francisco: Jossey-Bass.

D’Andrade, Roy. 1992. “Schemas and Motivation,” in Roy D’Andrade and Claudia Strauss, Human Motives and Cultural Models. Cambridge: Cambridge University Press, pp. 23-44.

Ghais, Suzanne. n. d. “Report on Action Evaluation of the Stara Zagora Multi-Ethnic Commission, Bulgaria, Baseline State.”

Kelman, Herbert. 1995. “Contributions of an Unofficial Conflict Resolution Effort to the Israeli-Palestinian Breakthrough,” Negotiation Journal,  11:19-27

Lederach, John Paul. 1997. Building Peace: Sustainable Reconciliation in Divided Societies. Washington, D.C.: USIP Press.

Lederach, John Paul. 1995. Preparing for Peace: Conflict Transformation Across Cultures. Syracuse: Syracuse University Press.

Lindbloom, Charles. 1959. “The Science of Muddling Through.” Administrative Science Review. 19: 79-99.

March, James G. and Herbert A. Simon. 1958. Organizations. New York: John Wiley.

Ross, Marc Howard. (forthcoming 2000). “Creating the Conditions for Peacemaking: Theories Of Practice In Ethnic Conflict Resolution.” Ethnic and Racial Studies, Vol. 23.

Ross, Marc Howard and Jay Rothman. 1999. Theory and Practice in Ethnic Conflict Management: Theorizing Success and Failure. Basingstoke and London: Macmillan.

Rothman, Jay. 1997. “Action Evaluation and Conflict Resolution Training: Theory, Method and Case Study. International Negotiation, vol. 2: 451-70.

Rothman, Jay. 1999a. “Goal Setting and Conflict Resolution in the Cyprus Consortium’s Initiative.” In Marc Howard Ross and Jay Rothman. Theory and Practice in Ethnic Conflict Management: Theorizing Success and Failure. Basingstoke and London: Macmillan, pp. 176-194.

Rothman, Jay. 1999b. “Applying Action Evaluation to Community and Neighborhood Initiatives.” Paper presented at the Fannie-Mae Foundation Research Roundtable. March 17, 1999. Washington D.C.

Verba, Sidney. 1961. Small Groups and Political Behavior. Princeton: Princeton University Press.