VII. Evaluating and Improving the Process of Understanding and Influencing Behavior

Background

Evaluation can be defined as gathering information to determine the effectiveness of activities, projects, and programs for the purpose of making decisions about them. Such decisions can be about how to modify an activity to make it more effective; whether to continue to support it with inputs of money, staff time, and other resources; or whether to use it as a model for other programs. The root word of “evaluate” is “value.” In evaluating a program we try to answer the question: How valuable is (or was) it for solving the problems it is (or was) designed to solve?

Evaluation involves assessing effectiveness. One good definition is that of Passineau (1975): “Evaluation is a process of collecting, weighing, and using information which is pertinent to making decisions about the merit of a program.” Another is that of Steelquist (1993): “Program evaluation is really just a matter of envisioning program outcomes, designing ways to reach those outcomes, and finding out if those outcomes have been achieved.”

On one hand, evaluation seems simple and familiar: you have goals and objectives, and you check to see how you are doing after some period of time. On the other hand, evaluation sometimes seems esoteric, like something only outside experts do. But if evaluation is demystified it is easy to see that it should be an integral part of any activity, project, or program.

This chapter briefly summarizes some of the voluminous literature on evaluation, especially that most relevant to understanding and influencing environmental behaviors.

Evaluation Throughout The Process

There is a logical link between evaluation and all other stages of the process of understanding and influencing behaviors in conservation - the assessment, planning, and implementation stages (see Fig. 4). This logical link exists because evaluation tries to answer the question: Did this activity, project, or program do what we thought it would do and wanted it to do? Evaluation is an integral part of the whole process (Jacobson, 1991; Rugh, 1992). “Ideally, evaluation should be conducted from beginning to end, providing feedback on all stages of the development, implementation, and outcome of a program” (Jacobson, 1991).

Robert Steelquist, in “Evaluation - Right from the Start,” argues that you can only answer the question: Did this work? if you have clear goals and objectives before beginning an activity, project, or program (Steelquist, 1993). If you don’t have clear objectives at the beginning of a program, evaluation is like trying to decide who is the best shot after the fact, when the marksmen hadn’t agreed on which target they were shooting at. One may say she was best because she hit the bullseye of a target, but the other may say he was better because he was aiming at a tree much farther away than the target, and hit it! Susan Jacobson (1991) writes that “... without stated objectives, it usually is difficult to determine whether the program is successful, how it may be improved, or to justify its accomplishments to administrators or funding agencies.”

The need for clear objectives right from the start seems to be the message coming from the best work on the evaluation of activities designed to influence conservation and natural resources management behavior (Pomerantz and Blanchard, 1992; Dietz and Nagagata, 1995; Jacobson, 1991; Rugh, 1992; Steelquist, 1993).

As discussed at the beginning of Chapter 2, we found that, unfortunately, conservation projects frequently jumped straight into implementation without much assessment or planning. Because of this, their goals are often broad and vague, making evaluation difficult, if not impossible. It is therefore perhaps not surprising that when project activities do not seem to be having the desired effect, project staff have to go back to earlier stages of the process - to assessment and planning - and start over more deliberately.

One important purpose for evaluating is to improve implementation (see Fig. 4). This kind of evaluation collects information about what is happening at the lower levels of the means-ends hierarchy. Are inputs of resources generating the kinds of activities expected and needed? Are those activities involving the actors needed to influence the key factors that motivate critical behaviors, as planned? If not, re-evaluation of the adequacy of the stages that led to implementation - assessment and planning - may be needed. In this way, evaluation is an iterative activity; information about how things are working is used to steer and guide the implementation stage of the process. It is useful for adaptive management of activities, projects, and programs. This is sometimes called formative evaluation: “evaluation that helps you understand your program while it is under way” (Steelquist, 1993).

Another important reason for evaluating is to find out if the ultimate goals and objectives of the activity were realized - to evaluate outcomes or impacts of the process (see Fig. 4). Impact evaluation requires information about what is happening at the upper levels of the means-ends hierarchy. Did changes in the key factors that influence behaviors lead to more sustainable behaviors? Did maintaining or changing behaviors have the predicted and desired effect on social and environmental conditions, moving them in the direction of USING sustainability? This is sometimes called summative evaluation: “evaluation that helps you understand the program after it has taken place” (Steelquist, 1993). This kind of evaluation is useful for deciding whether an activity would make an appropriate model for replication by others, and for making decisions about whether to continue supporting the activity.

Evaluation makes use of some combination of the methods and tools reviewed in Chapter IV. Feuerstein (1986) and Rugh (1992) provide excellent, user-friendly discussions of methods and tools for evaluation. Creative use of a variety of methods is needed to overcome the limitations of any single technique. Evaluation exercises often suffer from people telling evaluators what they think the evaluators want to hear rather than what is actually true. Research methods that rely on direct behavioral observation or other alternatives to direct questioning may be needed for effective evaluation.

The hypothetical example from Madagascar (Box 1) showed that interventions can have unexpected and undesirable effects. That is especially true if activities have been designed on the basis of assumptions about behavioral motivations and not based on adequate assessment and research. Periodic evaluation is needed to correct the course of programs experiencing such unwanted effects - to provide the “self-correcting processes” needed to ensure that programs are effective (Jacobson, 1991). Unexpected effects are not always bad, however; interventions may inadvertently do exactly the right thing. In such cases evaluation can detect those unanticipated successes, encourage practitioners to learn lessons from them, and reinforce successful activities.

The time lag between activities designed to influence natural resource management behaviors and actual behavioral changes can be long.

According to Fred Weber (1992), “the time lapse between USAID inputs/outputs and people-level impacts is often in the neighborhood of 5-15 years. Monitoring progress, therefore, requires a series of intermediate indicators.” It is nearly impossible at the start of a project to predict how long it will take to change behavior. Evaluation techniques should take into account the very gradual behavioral changes that are harbingers of more significant or widespread changes. Even if only a few individuals adopt a new environmentally sound practice during the project, such small changes should count as a measure of success (Wood and Wood, 1990).

The Marine Bird Conservation Project in Quebec incorporated an evaluation dimension that provides an excellent demonstration of the effectiveness of a behavior-centered approach. Kathleen Blanchard carried out a follow-up survey in 1988, six years after her initial survey of heads-of-households in 1981-82. The 1988 survey showed several “significant changes in local knowledge of wildlife law, attitudes toward hunting and regulations, and level of harvest of birds and eggs” (Blanchard and Monroe, 1990). Although it is only an indirect measure of behavior, the mean response to the question: What percent of families in your village harvest sea-birds and eggs? dropped significantly from about 76 percent in 1981 to 48 percent in 1988.

Evaluation has also demonstrated the effectiveness of the approach taken by the Golden Lion Tamarin Project in Brazil. The results of a follow- up survey in 1986 were compared with the results of the initial survey in 1984; the comparison “indicated significant changes in knowledge and attitudes of local Brazilian adults and students. Since no other activities or media events occurred in the area ... these changes can be attributed to two years of this project’s activities” (Dietz and Nagagata, 1995).

Evaluation And Participation

The need for all stakeholders to participate in the process of understanding and influencing conservation and natural resources management behaviors was discussed earlier. Because evaluation is an integral and essential part of this process, it is clear that participation in evaluation, or participatory evaluation, is needed. According to Feuerstein (1986), “Some traditional evaluation approaches have tried to make the people suit or ‘fit’ the evaluation methods. The newer approaches aim to make the methods suit the people and their situation. The approaches and technology are tailored to suit the real contexts of development programmes, and the abilities and technical levels of the participants. The collective name for such approaches and methods is participatory evaluation.”

Ford, Razakamanarina, and Randrianarisoa (1994) describe new uses for participatory rural appraisal (PRA) methods in participatory evaluation. They point out that “... in spite of dramatic growth in people-based planning, there has been little formal or systematic application of participatory tools for monitoring and evaluation.” They describe innovative uses of some standard methods and tools used in the assessment stage - such as maps, calendars, and matrices - to gather information for evaluation. Information gathered in an initial PRA exercise can be saved in a village log book; then, if the same methods are used again at a later time, the new information can be compared with the baseline, trends detected, and community action plans modified.

Norman Uphoff (1992) argues that monitoring and evaluation exercises are most useful when they use a participatory process. He writes, “As much as possible, joint monitoring and evaluation activities carried out with the involvement of both intended beneficiaries and project personnel, rather than by either group alone, are desirable.” Uphoff cites examples from both Tanzania and Thailand where government officials and villagers alike changed their behavior once participation in monitoring and evaluation exercises made them aware of village conditions.

Feuerstein (1986) suggests that “one of the reasons why it is difficult to show success or failure is that success or failure can mean different things to different people.” She gives an example of a project that built brick outhouses with locking doors in a rural village. Because even the houses in the village had no locks, outhouses came to be used for storing valuables like bicycles and tools - and were judged a great success by the villagers! They contributed little to village sanitation, the goal of the project, however, because they weren’t used as outhouses. Similar examples of divergent views of success could undoubtedly be found in conservation and natural resources management projects.

Whether the activity, project, or program had clearly defined objectives or not, outsiders can still look at it using their own criteria of value and judge its effectiveness from that point of view. They can come in after the fact, and using their evaluative lens, judge whether it had a certain valued outcome or not. But without some criteria or standards by which to judge effectiveness, whether built into the program from the start or imposed retrospectively, evaluation is impossible.

Evaluation And Hypothesis-Testing

Although the concept of sustainability has been around for some time, in many cases no one knows yet how it actually can be achieved. Our practical understanding of how to implement conservation is still growing, and in most cases a great deal of experimentation and hypothesis testing is needed to develop sustainable communities and societies.

Some people now talk about hypothesis testing in conservation and natural resources management projects. The USAID/Madagascar SAVEM Project does so, for example, as does the Biodiversity Conservation Network described in Box 16. The idea behind hypothesis testing in conservation is to gather information about what works while implementing activities.

Hypothesis testing in conservation and natural resources management is an excellent idea. It may, however, be difficult to test hypotheses and at the same time make use of formative evaluation to improve the implementation process. Evaluating the implementation stage of the process is useful for adaptive management - for modifying the intervention before its completion, if need be. But adaptive management is not really compatible with hypothesis testing. Hypotheses have to be falsifiable to be testable. If an intervention seems not to be working, and the goal is to test a hypothesis about what works, practitioners need to be willing to let it fail; otherwise they fail to carry out an experiment that tests the hypothesis.

The complexity of the situations in which natural resources management takes place leads to another difficulty for hypothesis testing. Bergdall (1993) points out that several factors may be causal and that “it is difficult to conclude ... that any one of them was the key determining factor over the others. Proof of a simple cause-and-effect relationship could not be found in the monitoring exercise. A review of the evidence does suggest that the planning seminars and subsequent follow-up work in Managhat and Endabeg [Tanzanian villages] played an important catalytic role in enabling the process of self-reliant development to occur.” The distinction between a catalyst and a cause may be useful, if evaluation methods can demonstrate such a catalytic role.

VIII. Conclusion

The analytical journey reflected in this report brought us face to face with many of the challenges of sustaining the natural resources base while meeting people’s needs and aspirations in Africa.

We came to see that people interact with their environment through their behavior - their decisions, practices, and actions. The behavior of individuals and groups forms the interface between ecological systems and human social systems; behaviors, therefore, provide a “window” into those systems. We realized that promoting sustainable natural resources management requires efforts to maintain some behaviors and change others.

We learned that each community and culture makes decisions about how to use the natural resources in its environment in the context of its own array of values. Successful conservation and sustainable natural resources management requires integrating the values and interests of a range of stakeholders and actors from all levels (local, national, and international), and that integration requires the active participation of stakeholders from all levels.

In our field work we learned that although the pivotal role of social factors in conservation and sustainable natural resources management is increasingly recognized, many conservation and natural resources management practitioners still lack practical ways of understanding and influencing environmentally relevant behaviors. A conceptual framework for setting goals and objectives, designing and implementing activities, and evaluating the effectiveness of those activities - a process for understanding and influencing behaviors - was one clear need. An-other need was for methods and tools for gathering the social information required to apply such a process effectively. We found, for ex-ample, that although some conservation practitioners were already using various methods and tools of social research, such as surveys and community meetings, many were not aware of the wide range of methods they could be using, especially participatory methods. Neither a process nor information-gathering methods alone are sufficient; both are needed.

In our field work we found that the first stage of the process of understanding and influencing behavior, which we call assessment, is often neglected. We found, for example, that activities designed to influence conservation and natural resources management behaviors are often based on untested, and sometimes mistaken, assumptions made by their planners and implementers. Assumptions about what motivates behaviors, or whether those behaviors are sustainable, are not often checked through social assessment, especially of a kind that involves real participation in the process by the actors themselves. Therefore, to meet a critical need, we focused our attention on the assessment stage of the process.

An understanding of the context and motivations of behaviors provided by assessment and research can be used to design and implement activities aimed at influencing those behaviors. Depending on which factors are key determinants of critical behaviors, different kinds of activities are needed. A number of general types of activities, each of which is most appropriate for influencing one or several of the factors that motivate behavior, are discussed in the report:

We learned also about the importance of dispute resolution: when stakeholders differ widely in their values, interests, and views about what should be done, resolving disputes may the most appropriate and feasible thing to do, at least as a first step.

There is a logical link between evaluation and each of the other stages of the process of understanding and influencing behaviors in conservation and natural resources management - the assessment, planning, and implementation stages. Integrating evaluation into the entire process of understanding and influencing behaviors is important.

Some scholars and practitioners express the view that only trained social scientists can, or should, do the social assessment needed to plan, implement, and evaluate conservation activities. However, many practitioners and communities lack the resources to hire trained social scientists to provide all, or even some, of the social information they need. We believe that conservation practitioners and natural resources managers can benefit from learning the process, and some basic methods and tools, of social assessment.

We hope that this report will stimulate discussion and provoke thought among the designers, implementers, and managers of conservation and natural resources management activities. We view the report not as the final word on this complex subject, but only another step in an ongoing analytical safari. As more practitioners become familiar with and use the kinds of processes and methods described here, refining and adapting them to their own unique situations, many lessons will be learned.

We hope that this report contributes to that process and that the findings and conclusions reflected in the report will be useful to practitioners seeking to understand what motivates the decisions, actions, and practices that affect the environment, and will thereby enable them to identify appropriate activities for influencing those behaviors in order to promote sustainable natural resources management and conservation. Ultimately we hope that this process enables practitioners and communities to maintain and improve both the quality of their lives and the health of their environment.