3. Lessons Learned About Hypothesis Testing Grant Programs

BCN is unique among donors in the conservation and development world in that we not only provides funds to field practitioners, but also work with these partners to explicitly test a hypothesis regarding the effectiveness of enterprise-oriented approaches to community-based conservation. As a result of this dual grant-making/hypothesis-testing role, the BCN program has different requirements than other grants programs. In particular, we work proactively with our partners and have an imperative need for detailed monitoring and evaluation data to be collected at each project site.

We believe that this grant-making/hypothesis-testing approach is a powerful and cost-effective strategy not only for using grant funds to achieve conservation and development objectives, but also to determine what works and why. The BCN's analytical approach needs to be considered with regards to future grants programming by BSP, USAID, and other donors in the conservation and development field who are interested in testing hypotheses. In the following sections we thus present some of the lessons we've learned over the past three years for those of you who might be interested in approaching grant making in a similar fashion. We present these lessons in four categories:

1. Program Design and Project Selection

2. Testing the Hypothesis

3. Structuring the Overall Program, and

4. Process Lessons.

For each point, we present both a brief discussion of the idea and an illustrative example from BCN's experience -- the things that have worked for us and the mistakes that we have made. This is a preliminary draft that we are presenting here in order to obtain feedback and encourage discussion amongst our partners. In the coming year, we hope to refine and expand this analysis into a more detailed document. We thus welcome your comments and responses.

3.1 Program Design and Project Selection

• Focus your program on a narrow range of stages and set of strategies for achieving conservation. Conservation of biological diversity is a complex and dynamic process that can take years to complete.² At any given project site, there is a broad continuum of stages that a project must pass through in order to achieve an adaptive management regime that ultimately promotes conservation. The specific stages that have to occur at any given site depend on local conditions. In general, however, these stages can be characterized as follows: 1) Promoting social organization, 2) Identifying problems and threats, 3) Continuing institutional organization, 4) Developing interventions to counter threats, 5) Monitoring the success of these interventions, and 6) Responding accordingly. Furthermore, within each of these stages, there are a number of strategies that might be employed to move the project through the stage. In designing a hypothesis testing grants program, you need to select the specific stage or stages that you will focus on as well as the specific strategies that you will consider. The "pace" of the project must be determined by the local people in response to local conditions. It is better to focus on a single stage and narrow set of strategies and do it well than to rush a community through multiple stages and types of strategies.

  Examples: 1) Early on, BCN staff realized that owing to our three-year grant window, we could not fund projects that were in the initial stage of developing relationships with the community. Instead, we needed to seek out projects where implementing groups had a long history of relationships with the community and could thus focus on developing interventions to counter threats and monitoring the success of these interventions. 2) Within our efforts on the interventions stage, we are not focusing on enterprise-oriented strategies for countering threats. We have deliberately chosen not to consider other strategies such as direct protection or environmental education.

• Maintain your focus, but enable your partners to consider other strategies. In any given project location, there is generally a wide variety of different strategies that can be undertaken to move through a given stage. Furthermore, these strategies are not necessarily mutually exclusive -- your partners may need to do both environmental education and direct protection. This creates a tension in an organization that is trying to both achieve conservation and test a hypothesis. On one hand, effective conservation at a given site often requires using many of these strategies. On the other hand, effective hypothesis testing requires that you focus on one or two strategies. As a result, while you should not abandon your focus, you should also encourage your partners to undertake the proper mix and sequence of strategies warranted by the conditions at their site.

  Example: A commonly heard misperception about the BCN is that we encourage our partners to focus only on enterprise-oriented approaches to conservation and ignore other approaches such as direct protection, indirect economic development, or environmental education. In reality, while we do focus on enterprise-oriented strategies for countering threats (as above), we do recognize the need to consider other strategies and we encourage our partners to find ways in which to do so.

• Develop analytical criteria for selecting a portfolio of projects as early as possible. A hypothesis-testing grants program needs to select a portfolio of projects that will enable it to complete a rigorous analysis of the ideas in question. You can most effectively undertake this selection process by identifying the program goals, hypotheses to be tested, and sets of key dependent variables as early as possible. Loosely defining an "ideal" mix of projects will ultimately facilitate the process of requesting proposals, selecting projects, and performing analysis of the results.

  Example: The relatively narrow focus of the BCN on enterprise-oriented strategies for conservation enabled us to bypass project proposals that did not have an enterprise linked to biodiversity at their core.

• Be prepared to solicit proposals to fill gaps in your portfolio of projects. A strictly "open-access" grant making process means that your portfolio will necessarily be drawn from the groups that choose to apply to you. In many cases, however, you may find that this pool of applicants is not broad enough to permit a full test of your hypothesis. You may thus need to proactively solicit proposals to fill gaps in your portfolio.

  Examples: 1) In BCN's early stages, we received almost no proposals from South Asia including especially India. We thus had to spend time soliciting proposals from this region. 2) Despite proactive efforts, BCN received very few proposals from private-sector entrepreneurs. As a result, our test of the BCN hypothesis will necessarily be somewhat restricted in its scope.

• Assist NGOs in project design. Many of the proposals you will receive may have interesting ideas that may not have been fully developed or that only partially fit within your program's criteria. If you have allocated time and staff resources to work with the project teams, you can often revise the projects to meet both your needs and theirs. A "project mapping" or "conceptual model"³ approach to project design can be an effective tool for working with teams to develop projects to meet all parties' needs.

  Example: The most consistent shortcoming across the nearly 500 proposals and concept papers reviewed by BCN staff was a breakdown between a project's stated goals and objectives and the set of interventions that were proposed to achieve them. BCN staff were able to work with projects that had already received Implementation Grants, but the program would have been greatly enhanced if we had the time and resources to work with prospective grantees prior to awarding the final grants.

• Integrate monitoring plans with project design. Project monitoring efforts will provide projects with both the information that they need to manage their projects and the bulk of the data that you will use to evaluate your hypotheses. A major constraint to monitoring, however, is that field-based staff implementing the project and monitoring plans may not always clearly understand what information is most needed or how to collect this information in an efficient manner. Integrating monitoring plans into the project conceptual model and workplan will help all project staff understand what data need to be collected.

  Example: At a BCN monitoring workshop, one local NGO staff member said: "In the past, the foreigners used to come in and tell us what things to do without telling us why these things are important. Now we fell like we can develop an understanding of why we are taking all these steps."

• Take risks on projects with strong leadership and the potential for good working relationships. In selecting projects, you should generally follow the criteria that you have developed. In some cases, however, it may be worth funding projects that do not fit all your criteria, particularly if the projects have strong leaders with whom you think you can develop a good working relationships. Strong leadership is characterized by a) credibility within the community, b) willingness to adapt to a challenging situation, and c) long-term vision of how the community should change to ensure a higher quality of life. It can often outweigh initial inadequacies of design.

  Example: Many of the BCN-funded projects that have provided the most interesting results are characterized by having strong and visionary leadership.

3.2 Testing the Hypothesis

• Value "failure" as much as "success." For typical grant making programs, when projects or specific components of projects fail to meet their objectives, they can only be written off as losses. For a hypothesis testing program, however, these "failures" serve an important purpose -- they become your controls. For ethical and moral reasons (similar to medical research with human subjects), it is not possible to initially select "deficient" projects to serve as experimental controls. However, when projects run into difficulties, you can learn as much from the challenges that they face as from the successes that they have.

  Example: The "Challenges" section of the "Stories from the Field" in Section 2 dramatically illustrates how valuable this information can be.

• Define areas in which projects in your "network" can collaborate with one another and with outside projects. One of the best sources of technical assistance for the projects that you are supporting are the other projects in your "network." You can effectively tap this knowledge by facilitating workshops and other exchanges of information about focused topics. In addition, you can also work with other projects outside of your network that are involved in similar issues -- these projects can also potentially be used to expand your "sample size."

  Example: BCN has focused its networking efforts on a narrow range of topics including a) monitoring plan development and b) enterprise skill building. This intense focus has been quite successful in assisting projects and enhancing BCN's understanding in both areas. In addition, we have been able to reach out to other groups working on similar issues.

• Keep the larger development context of an areain perspective. Often, aspects of the larger development context of the area, such as population densities and existing patterns of economic activity and prevailing agricultural practices, have serious implications for the success or failure of projects. These contexts should be clearly recognized in the project design, selection and evaluation.

  Example: BCN's adaptive management approach encourages partners to develop conceptual models that place their project in its ecological, social, economic, and political context.

• Be open to change. By definition, the process of adaptive management involves detecting and responding to changes in the biological, socioeconomic, and institutional context of the project. Allow your partners to be flexible.

  Example: BCN's adaptive management approach encourages partners to go through multiple iterations of their conceptual model to incorporate new information.

3.3 Structuring the Overall Program

• Establish a field presence as soon as possible. Getting your people in place in the locations where the projects will be working enables them to work with project partners on design and implementation and provides them with a enhanced understanding of local social and institutional contexts.

  Example: Relations with grantees and project implementation dramatically improved upon placing program officers in Manila, Jakarta, and New Delhi.

• Form a technical team with breadth and depth in key areas. Good teams don't just happen by chance. A grant making team of generalists is insufficient for evaluating proposals and adding value at project sites. Instead, you might aim for a team where each member's specific area of expertise (social sciences, ecology, cost analysis) is augmented by his or her working knowledge of a number of other areas. This complimentary mix of skills can ensure synergy and maximize team impact.

  Example: The BCN has attempted to develop a team whose members who have a "T" shaped map of skills -- depth in one or two areas and then a broad level familiarity in a number of other areas. This mix of skills has enabled BCN to provide a wide range of different types of technical assistance.

• Use an outside Review Group to augment staff capacities for proposal review. An outside Review Group that is sustained over the course of the grant-making organization's life provides multiple benefits including: 1) expertise on specific issues or places, 2) the credibility of an independent and unbiased review, 3) a sounding board that can comment on and give objective guidance regarding the overall design of the program, and 4) support in making difficult funding decisions. Note that in some instances, you may want to empower the Panel to make final decisions whereas in others, it may be more appropriate for them to serve in an advisory capacity.

  Example: The BCN Peer Review Panel met 5 times over the course of two years -- amazingly, 10 out of 12 members that attended the initial session were still in attendance two years later. Both BCN staff members and Peer Review Panels found the review process to be very valuable.

• Avoid staff turnover wherever possible. Losing your permanent staff has huge hidden costs. Staff knowledge of grantees and their ability to assist them is enhanced by the long-term relationship which develops. These relationships are strengthened if individual staff members are assigned responsibility for working with specific grantees. However, the departure of such staff imposes tremendous hidden costs associated with the loss of the institutional knowledge about the project, especially if there has been no mechanism for capturing this knowledge.

  Example: Institutionally, we lost a vast amount of knowledge about certain key grants when staff members moved on to other jobs. To some degree, this loss was ameliorated by detailed site visit reports that BCN staff are required to fill out.

3.4 Process Lessons

• Simplify the proposal writing process. National and local level NGOs and organizations often have great ideas for a conservation program. However, they are frequently thwarted from getting outside funding assistance due to often lengthy and complex proposal submission processes and their inexperience in writing proposals. Instead of issuing a standard call for complete proposals, it is probably more effective to solicit brief concept papers from potential applicants. Your field staff can then work with the candidate groups to further develop their proposal. This work not only reduces the quantity of information needed in a proposal, but is also of considerable assistance in the evaluation and grant-making process. Technical assistance can also be provided through this field presence at the proposal writing stage.

  Example: Our program started that we wanted to work on the grassroots level with community groups. At the same time, we required 60+ page proposals (before appendices!) with detailed business plans, biological and socioeconomic monitoring plans, and budgets that had to be written in English and submitted in duplicate to Washington DC with a fax number attached to them. Needless to say, it would have been easier for all concerned had we had our field team in place to follow the concept paper process outlined above.

• Specify the details of the partnership between grantee and grantor in clear terms. To avoid misunderstandings and disruptions in activities, it is important to clearly specify the nature of the partnership between grantor and grantee in terms of the roles and responsibilities of all parties. A particularly sensitive issue in hypothesis testing programs is the ownership and use of data collected about projects. Disputes can only be avoided if all parties agree to terms before the issues arise. Keeping in view the evolving nature of partnerships, you should periodically review and revise the terms of these relationships.

  Example: In the beginning, we had rocky relationships with a few groups over who would have control over and access to data. Over time, we have begun to iron these differences out.

• Maintain transparency for processes and decision making to keep expectations realistic. One of the biggest problems facing a grant making program is expectations. While soliciting ideas for projects; and deciding which projects will be funded, it is critically important that the expectations of local NGOs, people's organizations, and communities be kept realistic. Criteria for selection of projects and mutually agreed upon milestones need to be clearly communicated and transparent to all stakeholders.

  Example: The BCN Planning Grants were a double-edged sword. On one hand, they gave groups the resources needed to develop full-fledged Implementation Grant proposals. On the other hand, especially since more groups received Planning Grants than could get Implementation Grants, they created very strong expectations among partner organizations and community members. These expectations in some cases seriously damaged BCN's ability to work with these groups. The process would have perhaps been easier if we had followed the concept paper application idea outlined above.

• Insist that baseline monitoring data be collected BEFORE long-term funding is provided. Given all their other responsibilities (implementing activities, maintaining social relations, and dealing with bureaucratic requirements), it is often easy for project staff to let monitoring efforts slip -- monitoring is quite often the "incremental" activity that groups address after all the other work is started. Owing to the importance of monitoring to a hypothesis testing program, it is critical that you provide an additional "incentive" to ensure that monitoring data are collected while still providing the funds to do so.

  Example: Many BCN-funded projects have gone months or even years before finding the time to begin collecting monitoring data.

• Help national and local NGOs meet financial accounting rules. Often, one of the most difficult aspects of project management for national and local NGOs is managing the money that you give them. If you can provide assistance early on in dealing with the money, it will simplify your relations greatly in the future. Anticipating this technical assistance need and simplifying the reporting requirements can go a long way toward keeping the local partner's efforts focused on the conservation activities.

  Example: BCN experience in the field indicates that approximately 30 percent of the staff time in the first two years of the program was spent assisting NGOs in fulfilling their USAID financial reporting requirements. One of the most useful steps we have taken is to send our administrative staff to the field to work directly with our partners on financial systems.

• Use multiple sources of information to test your hypothesis. No single source of information (e.g. proposal, technical report) is sufficient to lend complete insight on evaluating complex conservation or development issues. Instead, it is best to triangulate from a number of different sources.

  Example: BCN staff use many sources of information including original proposals, technical reports, site visits, meetings, information from other donors and NGOs, newspaper articles, and the detailed monitoring information collected as part of the project by the grantees to enhance its understanding of the projects.

• Ensure that the duration of the grant is adequate for ensuring sustainability of the project. Donors generally like to see results as soon as possible. Most conservation efforts, however, require years if not decades of work to succeed. It is imperative therefore that your program be designed keeping the long-term nature of most conservation projects in view.

  Example: The BCN experience indicates that three to five years is too short a time to ensure either the financial viability of the enterprises or the adequate development of a community of stakeholders that would guarantee the sustainability of the conservation effort. Indeed, in most cases this time frame is too short to even get the basic monitoring frameworks functioning adequately so that the hypothesis can be tested.

---

²Although this section is written from the perspective of an organization with conservation goals, it can just as easily be applied to a group with economic development or other types of goals.

³See "Measures of Success by Margoluis amd Salasfky of BSP which is currently in press.

WHY BIODIVERSITY   FIELD STORIES   PARTICIPATE   RESULTS   MARKETSPACE ABOUT THE NETWORK   WHAT'S NEW   LEARNING MATERIALS   SEARCH   LINKS   SITE MAP   HOME