Appendix E: Guidelines for Assessing Impact

Appendix E: Guidelines for Assessing Impact

We will track the following USAID impact indicators:
SO-1 Area of habitat under effective management
SO-2 Documented improvements in conservation
IR1-1 Public awareness -- individuals participating in outreach
IR1-2 Public awareness -- decision-maker understanding
IR2-1 Strengthened polices -- index of policy results
IR4-1 Improved management -- site management benchmarks
IR4-2 Improved management -- number of people trained
IR5-2 Sustained financing -- enterprises
IR5-3 Sustained financing -- conservation funds leveraged
Each of the following pages provides the instructions from USAID as to how these indicators are to be assessed.

SO-1: Effective Biodiversity Conservation and Management -- Area

G/ENV/ENR Biodiversity Program

Strategic Objective:
Effective biodiversity conservation and management

Indicator 1:
Area of biologically important habitat under effective management
FY Area (ha)

Unit of Measure:
Area in hectares
97 Actual

Source:
Field visits and evaluations
98 Planned

Comments:
Two key conditions must be met for areas to be considered under effective management: (1) habitat quality is maintained/improved and/or the rate of habitat degradation is reduced; and (2) demonstrated institutional ability to monitor and respond to threats and opportunities (adaptive management)
Values are totals to date, i.e., cumulative.

Definitions
"Adaptive management" is a process of learning, defining policies and practices, and decision-making in which an organization responsible for the conservation of a particular ecosystem or area is responsive to the variations, rhythms, and cycles of ecological and social change and is able to react quickly and with appropriate interventions.
A 'site' is a specific geographical area where project activities occur involving a specific population of stakeholders over a determined length of time.
Calculation
These indicators are best calculated by first completing the index of site management benchmarks listed under IR4: Improve Management of Globally and Locally Significant Sites. The final indicators is then calculated by adding up the number of sites and summing the areas that have achieved adaptive management.
Example
A national level NGO is working with local landowners to conserve a 1500 square kilometer forest area. The land in the overall project is subdivided into 5 different areas controlled by clan members living in different villages. The rate at which the clans are adopting an adaptive management approach is highly variable across the 5 areas. The indicator summary thus treats the 5 areas as 5 separate sites and recognizes the different rates of progress expected over the next few years.
Using the data shown below in IR4, in 1996 two of the five villages have achieved adaptive management over the land that they control. The indicators are thus 2 sites and the sum of the land areas controlled by these two villages.

SO-2: Effective Biodiversity Conservation and Management -- Policy Improvement

G/ENV/ENR Biodiversity Program

Strategic Objective:
Effective biodiversity conservation and management

Indicator 2:
Documented improvements in biodiversity conservation as a result of strengthened policies or improved policy implementation
FY N

Unit of Measure:
Number of policy successes
97 Actual

Source:
Reports from partners
98 Planned

Comments:
Policies include laws, regulations, decrees, and agreements -- adopted by an organization -- which support the conservation and management of biodiversity. Policies can be designed and implemented at local, regional, national, and international levels.
Internal policies of conservation NGOs would not be included in this total.
Policy successes are documented examples where G/ENV-supported efforts to improve policies or policy implementation have directly contributed to on-the-ground biodiversity conservation.
Results are reported annually and as a cumulative total.

IR1-1: Increased Public Awareness and Decision-Maker Understanding of Biodiversity and its Conservation -- Outreach

G/ENV/ENR Biodiversity Program

Intermediate Result 1:
Increased public awareness and decision-maker understanding of biodiversity and its conservation

Indicator 1:
Number of individuals participating in G/ENV biodiversity outreach activities: workshops, networks, and oral presentations.
FY N

Unit of Measure:
Number of individuals (N)
97 Actual

Source:
Reports from partners
98 Planned

Comments:
Results are annual, not cumulative.

Definitions
"Outreach Activities" in this indicator only include formal meetings, workshops, etc. The indicator does not include informal site visits, stakeholder meetings, and other such activities.
Calculation
Count number of individuals participating in outreach activities.
Example
Stakeholders from the PNG project attended 2 enterprise development training sessions held in the provincial capital (6 individuals at the first and 9 and the second) and a monitoring workshop held another project site in PNG (5 individuals); the project total is 20 individuals.

IR1-2a: Increased Public Awareness and Decision-Maker Understanding of Biodiversity and its Conservation -- Publications and Media Events

G/ENV/ENR Biodiversity Program

Intermediate Result 1:
Increased public awareness and decision-maker understanding of biodiversity and its conservation

Indicator 2:
Number of project specific publications (P) sent to appropriate audiences, quantity of publications (Q) distributed, and number of mass media events (M).
FY P Q M

Unit of Measure:
Number of publications and mass media articles/stories.
97 Actual

Source:
Reports from partners
98 Planned

Comments:
Annual results will not be cumulative to avoid double-counting. For publications, the number distributed will be reported. For printed articles, the circulation will be reported. A subsample of the outreach materials will be audited to test the relationship between these materials and increased public awareness and understanding of biodiversity.
Results are reported annually and are NOT cumulative.

Definitions
"Project Specific Publications" are reports, brochures, and other documents generated about a project.
"Mass Media Events" are newspaper, radio, and television stories about the project or key impacts generated by the project.
Calculation
Score one point for each publication and media event.
Example
In FY96, the PNG project distributed 1000 copies of its annual report to relevant individuals in PNG and abroad. In addition, 100 copies of a poster describing conservation practices was given to schoolchildren in the project sites. These are counted as 2 publications. In addition, the project was featured in a local radio story, a BBC documentary, and project staff sent two letters which were published in the national newspaper. These are counted as 4 media events.

IR2-1: Strengthened National and Local Policies Supporting the Conservation and Management of Biodiversity

G/ENV/ENR Biodiversity Program

Intermediate Result 2:
Strengthened national and local policies supporting the conservation and management of biodiversity

Indicator 1:
Index of policy results
FY Score

Unit of Measure:
Number of publications and mass media articles/stores.
97 Actual

Source:
Policy Index
98 Planned

Comments:
Policies include laws, regulations, decrees, and agreements. They may be strengthened either by improving the policies themselves and/or their implementation.
The policy index is calculated by awarding 1 point for each step completed in each policy initiative: policy analysis, communication and educational activities to promote improved policies, improved policies adopted by national, regional, and local institutions, adequate implementation of these policies, and documented improvements in biodiversity conservation as a result of strengthened policies or improved policy implementation. One point is awarded for each organization that completes a step. The cumulative score is calculated for all steps completed for each policy initiative. The cumulative score is calculated for all steps completed for each policy initiative (see worksheet below).

Definitions
"Policies" include laws, regulations, decrees, and agreements. Policies can be designed and implemented at local, regional, national, and/or international levels.
"Policy Analyses" are formal or informal reviews of current regulations and changes that are needed to counter threats to the biodiversity of the region.
"Education Activities" involve sharing results of analyses with key decision makers responsible for formulating policies (e.g., members of a national legislature, a group of village elders).
"Policy Adoption" is when the policy is accepted by the group responsible for implementing the policy (e.g., ratified by a legislature, approved by a village council).
"Policy Implementation" is when at least one instance has occurred in which the policy has been used to counter a threat to the biodiversity of a site.

Index of Policy Results
- Sample Worksheet Policy Initiatives

Steps Completed 1 2 3 4

Policy analyses * * * *

Communication and educational activities to improve policies and their implementation * * *

Improved policies adopted by national, regional, and local institutions * * *

Improved policy implementation *

Documented conservation *

Sub-total 5 3 3 1

Total: 12

Calculation
The policy index is calculated by awarding 1 point for each step completed in each policy initiative: policy analysis, communication and educational activities to promote improved policies, improved policies adopted by national, regional, and local institutions, and adequate implementation of these policies. One point is awarded for each organization that completes a step. The cumulative score is calculated for all steps completed for each policy initiative. The cumulative score is calculated for all steps completed for each policy initiative (see worksheet above).
Example
The project in Papua New Guinea has four policy initiatives: 1) Working with the local clans to develop rules governing resource use in the forest area (e.g., no hunting of harpy eagle), 2) Working with the developing enterprises in the five villages to set uniform fees and rates for visitors coming to the project site, 3) Working with the provincial forestry ministry to ensure that timber concessions cannot be awarded in recognized conservation areas, and 4) Working with the National Government to settle a constitutional issue regarding the ownership of and access to subsurface mineral rights.
As shown in the table above, for 1996 the policies regarding resource use have been adopted and implemented. The policy regarding tourism rates and timber harvesting have been developed, but have not yet been implemented. Finally, in regard to mineral rights, only the policy analysis has been completed.

IR4-1: Improved Management of Globally and Locally Significant Sites -- Sites

G/ENV/ENR Biodiversity Program

Intermediate Result 4:
Improved management of globally and locally significant biodiversity sites

Indicator 1:
Index of site management benchmarks
FY Score

Unit of Measure
Site Index
97 Actual

Source:
Reports from partners
98 Planned

Comments:
Index of site management benchmarks calculated by awarding 1 point for each step completed for each site: local site assessments completed, change in legal status that favors conservation, management actions designed with appropriate participation, human and institutional capacity developed, management actions implemented, ongoing monitoring and evaluation established, adaptive management demonstrated.
The cumulative score is calculated for all steps completed for each site (see worksheet below). Not all sites would be expected to complete all these steps, as these vary by program.
Results ARE cumulative.

Definitions
A "site" is a specific geographical area where project activities occur involving a specific population of stakeholders over a determined length of time.
"Local Site Assessment and Threats Identified" refers to working with local partners to determine what courses of action that need to be undertaken to counter forces degrading biodiversity.
"Management Actions Designed" refers to developing the interventions that will be undertaken.
"Human and Institutional Capacity Developed" refers to providing the training and technical assistance necessary to undertake the actions developed above.
"Management Actions Implemented" refers to at least one step in the intervention undertaken.
"Change in Legal Status/Development of Policies" involves altering the ownership or control of a site to counter identified threats to the biodiversity of the site (e.g., gazetting as a park or protected area, providing tenure and resource use rights to local communities) or implementing a policy as defined above for IR2
"Monitoring and Evaluation" is defined as the periodic collection of information related to managing and measuring the impacts of project activities.
"Adaptive management" is a process of learning, defining policies and practices, and decision-making in which an organization responsible for the conservation of a particular ecosystem or area is responsive to the variations, rhythms, and cycles of ecological and social change and is able to react quickly and with appropriate interventions.
Calculation
Index of site management benchmarks calculated by awarding 1 point for each step completed for each site: local site assessments completed, management actions designed with appropriate participation, human and institutional capacity developed, management actions implemented, change in legal status made, ongoing monitoring and evaluation established, adaptive management demonstrated. The cumulative score is calculated for all steps completed for each site (see worksheet below). Not all sites would be expected to complete all these steps, as these vary by program.
Example
A national level NGO is working with local landowners to conserve a 1500 square kilometer forest area. The land in the overall project is subdivided into 5 different areas controlled by clan members living in different villages. The rate at which the clans are adopting an adaptive management approach is highly variable across the 5 areas. The indicator summary thus treats the 5 areas as 5 separate sites and recognizes the different rates of progress expected over the next few years. As shown in the table above, in FY96 only two of the sites have completed all of the steps. One of the sites has not completed any of the steps.

Index of Site Management Benchmarks
- Sample Worksheet Sites

Steps Completed Site 1 Site 2 Site 3 Site 4 Site 5

Local site assessment completed and threats identified * * *
*

Change in legal status that favors conservation *
*

Management actions designed with appropriate participation * * *
*

Human and institutional capacity developed * * *

Management actions implemented *
*

Change in legal status of site and/or development and implementation of policies that favor conservation *
NA

Ongoing monitoring and evaluation initiated *
*

Adaptive management demonstrated *
*

Point sub-total 8 3 7 0 2

Area of biologically important habitat (ha) 35,000 42,000 33,000 28,000 12,000

Total points: 20

IR4-2: Improved Management of Globally and Locally Significant Sites � People Trained

G/ENV/ENR Biodiversity Program

Intermediate Result 4:
Improved management of globally and locally significant biodiversity sites

Indicator 2:
Number of people successfully trained
FY N

Unit of Measure:
Number of individuals (N)
97 Actual

Source:
Reports from partners
98 Planned

Comments:
The "number of people successfully trained" is the number participating in training courses in which (a) a subsample of participants has been evaluated, and (b) the majority indicated that the training was useful and appropriate. Not all training needs need to be tied to the management of specific sites, but they are intended to ultimately improve efforts to conserve species and their habitats.
Results are NOT cumulative

IR5-2: Sustained Financing of Biodiversity Conservation -- Enterprises

G/ENV/ENR Biodiversity Program

Intermediate Result 5:
Sustained financing of biodiversity conservation through innovative public and private sector funding

Indicator 2:
Number of viable linked enterprises, gross value of these enterprises, number of people at the project site directly and indirectly employed by these enterprises, and the value of benefits received by stakeholders at the project site from these enterprises.
FY N $ B

Unit of Measure:
Cumulative number of enterprises (N), value in $US, and number of beneficiaries (B)
97 Actual

Source:
Reports from partners
98 Planned

Comments:
To be considered viable, conservation enterprises must meet two conditions: (1) they must be financially self-sufficient; and, (2) they adopt best business practices, which include monitoring the impacts of their activities on their natural resource base.
Results are reported annually and are NOT cumulative over years.

Definitions
"Linked Enterprises" are those which directly depend on biodiversity and thus provide incentives for biodiversity conservation.
"Viable Enterprise"¹ must meet two conditions: (1) they must be financially self-sufficient; and, (2) they adopt best business practices, which include monitoring the impacts of their activities on their natural resource base.
"Value" is defined as gross sales for the enterprise.
"Direct Employment" refers to people working for the core enterprise.
"Indirect Employment" refers to people working for other businesses that supply materials to or purchase goods from the core enterprise (e.g., selling vegetables to a tourism homestay, local people employed to transport products to regional markets).
"Number of Beneficiaries" refers to people receiving cash and non-cash benefits generated by the enterprises for local stakeholders (e.g., salaries, payments to community trust funds).
"Value of Benefits" refers to the value of quantifiable benefits described above.
Calculation
For each linked and viable enterprise, add up the value of the gross sales, the number of people directly and indirectly employed by the enterprise, and the total benefits generated by the enterprise.
Example
The project in PNG has the following summary data for the enterprises operating in each of the 5 sites within the overall project area.

Site 1 Site 2 Site 3 Site 4 Site 5 Total

Number of Viable Linked Enterprises 2 1 1 0 1 5

Gross Value of Enterprises 15,000 4,034 7,425 0 400 26,859

Direct Employment 42 16 28 0 6 92

Indirect Employment 112 19 3 0 15 149

Number of Beneficiaries 212 35 62 0 8 317

Value of Benefits 3,000 1,500 3,000 0 150 7,650

IR5-3: Sustained Financing of Biodiversity Conservation � Private Sector Funding

G/ENV/ENR Biodiversity Program

Intermediate Result 5: Sustained financing of biodiversity conservation through innovative public and private sector funding

Indicator 3: Conservation funds leveraged by G/ENV programs
Planned Actual

Unit of Measure: Cumulative funds leveraged (US$) 1995

Source: Reports from partners 1996

Comments: Leveraged funds may include matching funds and in-kind contributions. 1997

1998

1999

¹ Proposed revision to definition of 'viable enterprise': an enterprise that has reached the 3rd of 4 stages of development. Stage 1: enterprise is generating income; Stage 2: enterprise meets variable costs; Stage 3: enterprise meets variable and fixed costs; Stage 4: enterprise profitable while meeting all costs, including marketing and monitoring.

Index of Policy Results - Sample Worksheet	Policy Initiatives

Steps Completed	1	2	3	4
Policy analyses	*	*	*	*
Communication and educational activities to improve policies and their implementation	*	*	*
Improved policies adopted by national, regional, and local institutions	*	*	*
Improved policy implementation	*
Documented conservation	*
Sub-total	5	3	3	1
Total: 12

Index of Site Management Benchmarks - Sample Worksheet	Sites

Steps Completed	Site 1	Site 2	Site 3	Site 4	Site 5
Local site assessment completed and threats identified	*	*	*		*
Change in legal status that favors conservation	*		*
Management actions designed with appropriate participation	*	*	*		*
Human and institutional capacity developed	*	*	*
Management actions implemented	*		*
Change in legal status of site and/or development and implementation of policies that favor conservation	*		NA
Ongoing monitoring and evaluation initiated	*		*
Adaptive management demonstrated	*		*

Point sub-total	8	3	7	0	2
Area of biologically important habitat (ha)	35,000	42,000	33,000	28,000	12,000
Total points: 20

WHY BIODIVERSITY FIELD STORIES PARTICIPATE RESULTS MARKETSPACE

ABOUT THE NETWORK WHAT'S NEW LEARNING MATERIALS SEARCH LINKS SITE MAP HOME

G/ENV/ENR Biodiversity Program
Strategic Objective: Effective biodiversity conservation and management
Indicator 1: Area of biologically important habitat under effective management	FY	Area (ha)
Unit of Measure: Area in hectares	97 Actual
Source: Field visits and evaluations	98 Planned
Comments: Two key conditions must be met for areas to be considered under effective management: (1) habitat quality is maintained/improved and/or the rate of habitat degradation is reduced; and (2) demonstrated institutional ability to monitor and respond to threats and opportunities (adaptive management) Values are totals to date, i.e., cumulative.

G/ENV/ENR Biodiversity Program
Strategic Objective: Effective biodiversity conservation and management
Indicator 2: Documented improvements in biodiversity conservation as a result of strengthened policies or improved policy implementation	FY	N
Unit of Measure: Number of policy successes	97 Actual
Source: Reports from partners	98 Planned
Comments: Policies include laws, regulations, decrees, and agreements -- adopted by an organization -- which support the conservation and management of biodiversity. Policies can be designed and implemented at local, regional, national, and international levels. Internal policies of conservation NGOs would not be included in this total. Policy successes are documented examples where G/ENV-supported efforts to improve policies or policy implementation have directly contributed to on-the-ground biodiversity conservation. Results are reported annually and as a cumulative total.

G/ENV/ENR Biodiversity Program
Intermediate Result 1: Increased public awareness and decision-maker understanding of biodiversity and its conservation
Indicator 1: Number of individuals participating in G/ENV biodiversity outreach activities: workshops, networks, and oral presentations.	FY	N
Unit of Measure: Number of individuals (N)	97 Actual
Source: Reports from partners	98 Planned
Comments: Results are annual, not cumulative.

G/ENV/ENR Biodiversity Program
Intermediate Result 1: Increased public awareness and decision-maker understanding of biodiversity and its conservation
Indicator 2: Number of project specific publications (P) sent to appropriate audiences, quantity of publications (Q) distributed, and number of mass media events (M).	FY	P	Q	M
Unit of Measure: Number of publications and mass media articles/stories.	97 Actual
Source: Reports from partners	98 Planned
Comments: Annual results will not be cumulative to avoid double-counting. For publications, the number distributed will be reported. For printed articles, the circulation will be reported. A subsample of the outreach materials will be audited to test the relationship between these materials and increased public awareness and understanding of biodiversity. Results are reported annually and are NOT cumulative.

G/ENV/ENR Biodiversity Program
Intermediate Result 2: Strengthened national and local policies supporting the conservation and management of biodiversity
Indicator 1: Index of policy results	FY	Score
Unit of Measure: Number of publications and mass media articles/stores.	97 Actual
Source: Policy Index	98 Planned
Comments: Policies include laws, regulations, decrees, and agreements. They may be strengthened either by improving the policies themselves and/or their implementation. The policy index is calculated by awarding 1 point for each step completed in each policy initiative: policy analysis, communication and educational activities to promote improved policies, improved policies adopted by national, regional, and local institutions, adequate implementation of these policies, and documented improvements in biodiversity conservation as a result of strengthened policies or improved policy implementation. One point is awarded for each organization that completes a step. The cumulative score is calculated for all steps completed for each policy initiative. The cumulative score is calculated for all steps completed for each policy initiative (see worksheet below).

G/ENV/ENR Biodiversity Program
Intermediate Result 4: Improved management of globally and locally significant biodiversity sites
Indicator 1: Index of site management benchmarks	FY	Score
Unit of Measure Site Index	97 Actual
Source: Reports from partners	98 Planned
Comments: Index of site management benchmarks calculated by awarding 1 point for each step completed for each site: local site assessments completed, change in legal status that favors conservation, management actions designed with appropriate participation, human and institutional capacity developed, management actions implemented, ongoing monitoring and evaluation established, adaptive management demonstrated. The cumulative score is calculated for all steps completed for each site (see worksheet below). Not all sites would be expected to complete all these steps, as these vary by program. Results ARE cumulative.

G/ENV/ENR Biodiversity Program
Intermediate Result 4: Improved management of globally and locally significant biodiversity sites
Indicator 2: Number of people successfully trained	FY	N
Unit of Measure: Number of individuals (N)	97 Actual
Source: Reports from partners	98 Planned
Comments: The "number of people successfully trained" is the number participating in training courses in which (a) a subsample of participants has been evaluated, and (b) the majority indicated that the training was useful and appropriate. Not all training needs need to be tied to the management of specific sites, but they are intended to ultimately improve efforts to conserve species and their habitats. Results are NOT cumulative

G/ENV/ENR Biodiversity Program
Intermediate Result 5: Sustained financing of biodiversity conservation through innovative public and private sector funding
Indicator 2: Number of viable linked enterprises, gross value of these enterprises, number of people at the project site directly and indirectly employed by these enterprises, and the value of benefits received by stakeholders at the project site from these enterprises.	FY	N	$	B
Unit of Measure: Cumulative number of enterprises (N), value in $US, and number of beneficiaries (B)	97 Actual
Source: Reports from partners	98 Planned
Comments: To be considered viable, conservation enterprises must meet two conditions: (1) they must be financially self-sufficient; and, (2) they adopt best business practices, which include monitoring the impacts of their activities on their natural resource base. Results are reported annually and are NOT cumulative over years.

G/ENV/ENR Biodiversity Program
Intermediate Result 5:	Sustained financing of biodiversity conservation through innovative public and private sector funding

Indicator 3:	Conservation funds leveraged by G/ENV programs		Planned	Actual
Unit of Measure:	Cumulative funds leveraged (US$)	1995
Source:	Reports from partners	1996
Comments:	Leveraged funds may include matching funds and in-kind contributions.	1997
		1998
		1999