Biodiversity Support Program
Greater Than the
Sum of Their Parts:
Designing Conservation and Development
Programs to Maximize Results and Learning
A Practical Guide for Program Managers and Donors
TABLE OF CONTENTS
What Is a Typical Results-Oriented Program?
What Is a Learning Program?
Structure of This Guide
Adaptive Management and Learning Organizations
Step A: Develop Program Concept & Structure
Conduct Initial Situation Assessment
Develop an Explicit Model of What You Want to Learn
Clarify Overall Program Structure
Develop Program Goals, Objectives, and Activities
Set Up Program Monitoring
Step B: Select a Focused Portfolio of Projects
Develop and Circulate a Formal Request for Proposals
Screen Concept Papers
Determine the Criteria for Evaluating Proposals
Select Your Portfolio
The Science of Developing Valid Criteria and the Art of Developing Meaningful Criteria
Step C: Develop an Analytical Framework
Determine Your Key Audiences
Develop Conceptual Models of Projects
Combine Models to Identify Key Questions
Determine Data Collection Needs and Methods
Develop Data-Sharing Plans and a "Social Contract"
Step D: Implement Projects & Analytical Framework
Ensure That All Roles Are Covered
Invest in Face-to-Face Meetings
Step E: Analyze Data & Communicate Results
Compile Data in a Standardized Format
Analyze Data on an Ongoing Basis
Develop Creative Communications Products
Survey Audiences
Iterate
What are General and Yet Non-Trivial Guiding Principles
Costs of Learning Programs
Benefits of Learning Programs
Balancing the Costs and Benefits
The Future
The basic unit of conservation and development work is a project. A collection of projects being undertaken by a group is a program.
What Is a Typical Results-Oriented Program?
A typical results-oriented program involves implementing or funding a group of projects that are loosely clustered around some theme to achieve a conservation and development goal. For example, you might fund projects in a certain geographic region or academic discipline. Or you might develop projects near a specific national park or dealing with a certain policy issue. The key point here is that each project in your program is selected more or less independently of the others and, thus, there are few if any synergies. Suppose you fund 20 projects and 12 of them are completely successful while 8 are complete failures. The net benefit of the program is the sum of the impacts of the 12 successful projects plus any capacity developed in the organizations implementing the projects. A results-oriented program's net impact is at best the sum of its parts.
A Note About Terminology:
In this guide, we use the following terms:
What Is a Learning Program?
A learning program is a special kind of program that has two types of goals. The first involves achieving specific conservation and development objectives. The second involves systematically learning from your actions to determine what works, what does not work, and why. Under a learning program, a group of projects is selected to deliberately test a specific concept or set of hypotheses. For example, you might look, as the Biodiversity Conservation Network (BCN) did, at the conditions under which an enterprise- based approach to conservation is effective. Or you might try to determine what are the most effective strategies for conservation efforts in a country where you are working.
The key point here is that each project is selected to be part of a portfolio. This portfolio should be designed in such a way that it enables lessons to be learned by comparing the projects to one another. In this case, if we have funded 20 projects and 8 of them fail, these 8 failures are no longer wasted. Instead, they provide important information regarding our hypotheses. Indeed, we may learn as much or more from the failures as from the successful projects. Furthermore, the projects in the portfolio can readily exchange ideas and experiences. A learning program's net impact thus becomes far greater than the sum of its parts.
Another Term:
Structure of This Guide
In this guide, we discuss the steps involved in developing and implementing a learning program. These steps are outlined in the diagram on the next page. We then discuss some of the costs and benefits that are involved in using this approach.
These steps should generally be undertaken in a sequential manner as indicated by the large arrows in the diagram going from left to right. A key premise behind this diagram, however, is that the activities and products of each step are highly interconnected. Furthermore, although the general flow of the program is sequential from left to right as illustrated in the diagram, there is also an iterative feedback process (represented by the curved arrows on top of the diagram and the arrow on the bottom) between the steps. Thus, although these steps are presented in a highly structured fashion, we realize that most program teams would not follow such a restricted step-by-step order. Instead, an experienced program team would "work the problem from both ends"- thinking first about what information its audiences might want, considering which projects are likely to be included in the portfolio, and then maybe finally going back to the program design. So, although for clarity we have presented the approach as a linear process, we would encourage you not to feel bound to it.
A Final Note About Terminology:
Throughout this guide, we use the words "you" and "your" to refer to the reader, who we are assuming to be involved in managing or implementing a learning program. In many ways, however, this use of the word "you" should be interpreted more broadly as referring to all of the people involved in designing and implementing a program. In particular, while Steps A and B must necessarily be undertaken by a limited group of people, Steps C through E should be undertaken by everyone involved in the program and its component projects.
We've also written this guide as if you were starting at the very beginning of your program, so we assume that you will go through the entire process outlined in this guide from start to finish. This process, however, can be used almost as easily to reconfigure an existing program. You can use it to help think about your program in a structured fashion or even to help you understand or evaluate another program.
Finally, please keep in mind that program design is as much an art as it is a science. By this we mean that there is no one right way to do things. Instead, many of the procedures involved in this approach require balancing our guidelines with your beliefs and experiences. We encourage you to adapt the process presented here to meet your specific needs.
Examples From the BCN Experience:
The main text of this guide presents an idealized process for setting up and implementing a learning program. In the remainder of this guide, we use the sidebars and text boxes to provide specific examples of the process that BCN used. We also comment on ways in which we could have improved this process.
|
Adaptive Management and Learning Organizations The concept of a learning program is based on the principles of adaptive management and the learning organization. As outlined by Margoluis and Salafsky (1998), people in a number of different fields have more or less independently arrived at the concepts behind adaptive management. Adaptive management involves integrating program design, management, and monitoring to provide a framework for testing assumptions, adaptation, and learning.
Ecosystems Management
Business Management and Organizational Theory
Conservation and Development Project Management
|
STEP A: DEVELOP PROGRAM
CONCEPT & STRUCTURE
The first step in developing a learning program is to determine what problem or question you want to address with your program, what your specific hypothesis is, how you will structure your program, and how you will monitor it over time.
Conduct Initial Situation Assessment
Your first task is to assess the situation in which the program is trying to operate and determine what general problems and questions you want to address. In many cases, you probably already have a pretty good idea of what system and topics you want to address. If you do not, however, you may wish to conduct an assessment that involves answering variations of the following questions:
Develop an Explicit Model of What You Want to Learn
Once you have a general idea of what you want to address, your next task is to develop the basic hypothesis or hypotheses that you want to test. A hypothesis is generally best expressed in the context of a model of the system in which you are working. A model can be made out of words, mathematical equations, computer code, or pictures (see Margoluis and Salafsky 1998 for an example of how to develop graphic conceptual models). Whatever method you use, the key is to make your model in a way that everyone involved with your program can understand and discuss it.
Once you have created your basic model of the system, you can then use your model to explicitly state your hypotheses. It may seem a bit strange to be forcing yourself to write down hypotheses before you have done any real work — you may feel that you do not yet know enough. But this is exactly the point — by stating your hypotheses now, you can then come back and check and see whether you were right or wrong, and change it if necessary. By explicitly stating your hypotheses, you can also check your ideas with your colleagues and partners and make sure that everyone else working with you has a similar understanding of the situation and the changes to be made.
Testing the Enterprise Hype:
In BCN's case, our founders recognized that there was a lot of hype about the potential of enterprise-based aproaches to conservation, but that there had been little or no systematic efforts to study whether this approach would actually work. Our parent organization, the Biodiversity Support Program (BSP), had substantial experience in helping the United States Agency for International development (USAID) design and implement grant-making programs. It thus seemed like a natural fit to establish a grant-making program as part of BSP that would test the hypothesis outlined in the next box.
BCN's Core Hypothesis:
In BCN's case, our hypothesis about enterprise-based approaches to conservation stated:
If enterprise-oriented approaches to community-based conservation are going to be effective, the enterprises must:
In effect, the hypothesis is that if local communities receive sufficient benefits from an enterprise that depends on biodiversity, then they will act to counter internal and external threats to that biodiversity. (A graphical depiction of this hypothesis in relation to other conservation strategies is shown on the next page.)
A Graphical Model of the BCN Hypothesis
The BCN core hypothesis (Model 3) can perhaps
be best understood in relation to models of two other conservation strategies
(Models 1and 2). In all models, the target condition is the biodiversity
of the project site. Internal threats are the result of activities by
local stakeholders. External threats are the result of activities by
outside parties. In these models, solid rectangles represent states of the system,
dashed rectangles represent intermediate effects, and hexagons represent project
activities.
Model 1: Protected Areas - Under this model, the project team establishes
a protected area to stop both internal and external threats to the biodiversity
of the project site.
Model 2: Economic Substitution - Under this model, the project team
promote an economic activity as a substitute to damaging activities by the local
stakeholders. Examples might include growing coffee in a buffer zone or setting
up a shoe factory.
Model 3: Linked Enterprise - Under this model, the project team develops
an enterprise that is directly linked to the biodiversity. This enterprise provides
benefits to a community of stakeholders who have the incentive and capacity
to counter the internal and external threats to the biodiversity.
Clarify Overall Program Structure
The next task is to outline the basic structure of the program. As a rule, you will probably already have a pretty good idea of the answers to the following questions. But it can be helpful to make these answers explicit.
The BCN Structure:
BCN was established as part of the Biodiversity Support Program, a consortium of World Wildlife Fund, The Nature Conservancy, and World Resources Institute. Initially, BCN was set up as a five-year program, but was later extended to seven years to complete activities.
BCN was set up with only a director and two other staff members. It was expected that this group would be able to develop a request for proposals, review the proposals, and then provide the funding. The idea was then to add a few more staff over time. Based on the initial set of proposals that we received, however, we quickly realized that we had to rethink this strategy and devote far more resources to proactively working with our grantee partners to develop proposals. BCN was ultimately expanded to as many as 15 full-time program and administrative staff housed in small "satellite" offices in four different countries.
BCN was established as a competitive grants program. Two types of grants were awarded: Planning Grants up to $50,000 for a 6 to 12 month period to do on-site assessments and develop a full-fledged proposal; and Implementation Grants for a 3 to 4 year time period to implement proposed activities. Proposals were screened by BCN staff and an outside review panel. We then obtained concurrence for the proposals that were selected from USAID.
BCN received $20 million form USAID. Approximately $12 million of this went directly to grant funds while the remainder was used to administer the program.
Develop Program Goals, Objectives, and Activities
Your next task is to develop an overall management plan including goals, objectives, and activities for the program (see Margoluis and Salafsky 1998 for a detailed discussion of these items). Note that compared to projects, goals and objectives for programs tend to be more process-oriented than impact-oriented.
BCN's Goals and Objectives:
BCN's goals were to both (1) support enterprise-based approaches to conservation at a number of sites across the Asia/Pacific region, and (2) test the effectiveness of these approaches and provide lessons learned to key clients and audiences. Key objectives and activities were tied to each of Steps A through E in the program. In addition, we also had two crosscutting objectives that involved developing the skills of our project partners and enhancing the development of partnerships between different grantees.
Set Up Program Monitoring
The final task is to determine how you will monitor your overall program. Although monitoring requires an investment of time and money, this investment can save resources in the long run by ensuring that the program is effective in reaching its goals and objectives. Monitoring also enables you to take corrective action as it becomes necessary and to demonstrate to outsiders that your work is worthwhile.
Monitoring is generally most effective if it is built into the program design from the start. Although most monitoring work comes later in our process, it is important even at this point to have a good idea of what information you will need and how you will go about getting it. Specific questions that you may wish to address include:
BCN's Internal and External Monitoring:
We evaluated the BCN program in relation to our stated goals and objectives. Internal monitoring was conducted every six months as part of our standard reporting requirements to USAID. A mid-term evaluation was conducted by an outside team. We used the results to continually adapt the program.
STEP B: SELECT A FOCUSED PORTFOLIO
OF PROJECTS
The second step in developing a learning program is to establish and implement the specific process that you will use for reviewing and selecting the projects in your program. This process can be highly systematic or it can be on a completely ad hoc basis. For example, if you are a donor running a worldwide grants program, you may have to spend a good deal of time on this step. If, however, you are a program coordinator running a small program with only a limited number of projects to choose from, you may be able to go through this step fairly quickly. The key point is that all participants in the process - including your prospective grantees or partners - should have the same understanding of how the process works.
To this end, it is often very helpful to develop a flow chart outlining how proposals will be selected (see example on the next page). It is also helpful to develop and distribute an explicit statement of the criteria that you will use to evaluate proposals. This means you may have to undertake this task while you are developing your request for proposals. Finally, since you will be using this portfolio of projects to test your hypothesis, it is useful to review the tasks in Step C prior to completing Step B.
Develop and Circulate a Formal Request for Proposals
Your first task in this step is to develop a request for proposals (RFP). Your request for proposals should outline the overall purpose of your program, the specific requirements that you are setting up, and the format in which people should submit proposals. As discussed in the sidebar, it is generally better to avoid having people send unsolicited, full-fledged proposals. Instead, it is usually far more effective to have people submit two to three page concept papers that outline:
This concept paper should be reviewed by your program staff, who may also wish to talk directly with the prospective applicants to further develop their ideas. Once a concept paper has been accepted, the grantee can then prepare a complete proposal. In reviewing and commenting on concept papers, it is important not to "read more into the proposal" than is actually there. There is a fine line between helping people to draw out and communicate their ideas and imposing your ideas on them.
BCN's Two RFPs:
Based on our initial request for proposals (RFPs), BCN initially accepted full-fledged Implementation Grant proposals as well as Planning Grant proposals. We soon realized, however, that we could save both potential grantees and ourselves time and effort if we first asked for smaller, more focused Planning Grant proposals or even just concept papers. We thus issued a revised RFP.
We also found, however, that in asking for concept papers and Planning Grant proposals, we created a tension between helping people to improve their proposals and creating false expectations on the part of the potential grantee. To this end, you should inform potentioal grantees that an invitation to submit a full proposal - or even a discussion with a program officer about their proposal - is not a guarantee of future funding.
Overview of the BCN Planning Grant Review Process
The process that BCN used to evaluate grant proposals changed over time as well learned from our experiences. The following flow chart illustrates an "idealized" version of the process we used.
Screen Concept Papers
Once your request for proposals has been published, you should start to receive concept papers and other inquiries. All but the most casual inquiries should be logged into a database. Concept papers should then be reviewed by one or two of your program staff members. Those concept papers that obviously do not meet your initial filters should be turned down. All other proposals should be entered into the system. Examples of initial filters that you might want to use include:
BCN's Initial Filters:
BCN only accepted proposals from a limited number of countries that were defined by USAID when the program was first established. We also, of course, only focused on enterprise-based approaches to conservation.
For programs that are trying to work with projects in developing countries and other arenas where people have less experience with proposal writing, your program officers may have to work proactively with people who have promising ideas to help them develop their ideas, concept papers, and proposals. However, as noted earlier, you need to make sure that you are not creating false expectations.
Helping Groups Get Over the Bar:
BCN had relatively strict proposal requirements, although we often loosened these requirements when dealing with non-native English speakers or groups that were unfamiliar with proposal writing. Our strategy here, however, was not to "lower the bar" but to invest staff time or resources in working with the project teams so that they could then "get over the bar on their own."
Determine the Criteria for Evaluating Proposals
Before you can fully evaluate project proposals, you need to develop a set of criteria that you can use in your evaluation process. These criteria must be made explicit so that all reviewers are evaluating proposals by the same measures. Developing good criteria is one of the most critical steps in this whole process. This section presents a number of common criteria that you may want to think about using in your review process. Furthermore, in the boxes on the following pages, we discuss the science and the art of developing criteria.
Basic Criteria
These criteria are used to prescreen the proposal before it reaches the full review panel. Examples of basic criteria that you might want to use include:
| The Science of Developing
Valid Criteria...
Rules for a Valid Criterion Criteria are formal decision rules that determine which projects should be included in your portfolio, and, just as importantly, which should not. Each criterion should outline a specific set of categories and then specify which categories are included in the portfolio and which are not. Valid criteria possess the following characteristics : 1. The criterion is bounded - the edges of the set are sharply defined. 2. Categories are of the same taxa - all of the categories are of a uniform type. 3. Categories are discreet and exclusive - the edges of the categories are sharply defined and the categories do not overlap. 4. Categories are comprehensive - the categories completely fill the area of the set. It is perhaps easiest to illustrate what makes a given criterion valid through the use of a simple example. Let's assume that you have a budget of $50,000 dollars and 12 proposals for projects (P1 to P12) that will cost $10,000 each. The projects come from a variety of different countries in Asia that were eligible for BCN funding as shown in the following table.
One criterion for selecting which projects to undertake might be geographic distribution. In this example, you can think of the overall criterion as being the set of all countries in Asia eligible for BCN funding (the shaded box) and each potential country being a category (white circles) in the set as shown in Example 1. If your decision rule is that you want to get balance across the categories in this criterion, then you might decide to select roughly one project in each of the country categories as shown in the diagram. In this example, it is clear that the criterion encompasses only a specific set of Asian countries eligible for BCN funding, that each of the categories represents a different country, that any observer could assign a proposal to one and only one bucket in the box, and finally that there is no empty space within the set of the criterion. Note that is okay if the Thailand category is empty- that there is no proposal to put into it. What is important is that a category exists for every potential proposal within the space of the criterion. Example 1.A Valid Criterion
Example 2. An
Invalid Criterion The second example, on the other hand, violates all of the rules for a valid criterion. It is not clear where the right edge of the box is located and which countries are in and which are out. Two of the categories contain items other than countries, and it is not clear what would go into them. A proposal from New Delhi, for example, could go into its own category, into the India category, or into the South Asia category, making allocation difficult. Finally, no category exists for every potential proposal within the criterion - if a proposal from Thailand were to come along, there is no place to put it. Using Multiple Criteria A good portfolio generally uses multiple criteria. For example, in addition to geography, you may wish to pick only those projects that have good leadership. Let's assume that the numbers 1 to 12 represent a ranking of the leadership of the project teams where 1 is the best and 12 is the worst. You might thus create a second criterion that involves setting up four categories of leadership ability (I to IV). Using only this criterion, the portfolio selection might look as follows:
If however, we wanted to combine this criterion with the geographic one, our selection process suddenly becomes more complicated. This process is easiest to show using a table. In this table, one criterion (geography) forms the column headings and the other (leadership) forms the row headings. Each of the two criteria are valid in that they follow the rules and thus each cell is a category in both a row and in a column.
To decide which projects to select in this case, you can move across the columns from left to right until your budget is exhausted. First, however, you need to decide which criterion is more important. If it is more important that you have geographic balance in conjunction with the best available leadership, you would select the projects in this order: { P3, P1, P2, P9, P4 } If, however, it is more important to have high quality leadership with as much geographic distribution as possible, then you would select: { P1, P2, P3, P4, P6 } In either case, if you are not restricted to making decisions at this particular time from this particular list of projects, you might want to proactively solicit or develop projects from Thailand to obtain better geographic balance. The above example shows how balancing portfolio considerations works across two dimensions. To increase this to more criteria, you merely add additional dimensions to the table by adding additional valid criteria. This process gets complicated to show visually. However, you can easily do this by creating a table placing each criterion in a column and the candidate projects in rows. You can then assign points to each project under each criterion or rank the projects relative to one another (see Margoluis and Salafsky 1998 for more detailed discussions of matrix ranking techniques). The relative priority of the different criteria can be addressed by assigning weights to each column or by sequentially evaluating the columns. |
| ...and the Art
of Developing Meaningful Criteria
The previous text box outlined how to develop valid criteria by using a simple set of rules. However, just because a criteria is valid does not mean it is meaningful. For example, one criterion that you might develop would be to sort projects by the last letter in the name of the group implementing the project. This criterion is valid - it satisfies all four rules. However, it would obviously be pretty silly to develop a portfolio of projects whose names end in "g" or "r". Developing meaningful criteria is where the art of this entire process comes into play. We can't give you any hard and fast rules for this part. The specific criteria that your group chooses to deem meaningful depends on what hypotheses you are trying to test and who you are. Indeed, to borrow a concept from Robert Pirsig (1974), it is not so much that your group is determining what is meaningful as much as your group defines itself by what it chooses to view as meaningful. For example, a program being implemented by a government agency might choose as one of its "meaningful" criteria to have projects more or less equally distributed among all the provinces in the country. An environmental NGO, by contrast, might care less about political geography and instead choose as its meaningful criterion sites where there are local leaders interested in implementing and testing locally managed protected area strategies. Neither group is right or wrong - they just have different interests and needs. |
Criteria Pertaining to the Project's Merits
These criteria are used to assess the basic merits of the proposal. Examples of criteria that you might want to use include:
An Emphasis on Results:
Because of our conservation impact goal, an important criterion for BCN was whether the project was taking place in a globally significant area of biodiversity.
Criteria Pertaining to the Group's Capacity
These criteria are used to assess the basic merits of the proposal. Examples of criteria that you might want to use include:
The Most Important Criteria That We Should Have Considered:
In retrospect, perhaps one of the most important criteria that we should have used was the ablility of the group to engage in self-reflection. Many project teams are either unwilling to critically examine themselves or do not have an innate curiosity, both of which are required for effective hypothesis testing.
In addition, when providing funding for projects, it is important to try to identify those groups who are genuinely interested in the focus of your portfolio, as opposed to those who are merely trying to design a project that will get funded.
