How to structure a policy coherence assessment?
Introduction
A policy coherence assessment can be structured into four phases: 1) structuring the assessment, 2) data collection, 3) data analysis and 4) synthesis and conclusions. The table below provides a quick overview of the four phases, which are described in more detail in the guidance section. This approach follows the European Environment Agency’s guidance for policy evaluations (EEA 2017) and is in line with the European Commission's Better Regulation guidelines.
| Phase of the assessment | Steps to be undertaken |
|---|---|
| 1. Structuring the assessment | Preparing an initial overview of policies to be evaluated (step 1.1) |
| Setting the time frame (step 1.2) | |
| Determining the geographical scope (step 1.3) | |
| Defining the evaluation questions (step 1.4) | |
| 2. Data collection | Data needs |
| Data collection methods | |
| 3. Data analysis | Policy coherence analysis |
| 4. Synthesis and conclusions | Answer evaluation question(s) |
Guidance
Phase 1: Structuring the assessment
The structuring phase defines the boundaries of the assessment in terms of policies, time frame and geographical scope, and determines the evaluation questions to be addressed.
In this phase, you need to select the policies that will be part of the assessment. Include all relevant policies from the governance level(s) that are part of the assessment. Depending on whether you are looking at horizontal or vertical coherence, this may be one or more governance levels. Which policies are relevant will depend on what you are interested in finding out. What is important is that there is some level of interaction between the policies you select. These could be policies within one policy area (for example fisheries), or policies from different policy areas between which there are impacts and dependencies (for example agriculture and water management).
You also need to set the time frame for the assessment. Determining the time frame of the assessment is important to clarify which versions of policies are being assessment and which policies are included or excluded. This is particularly relevant for policies that are under revision or when new proposals are announced. When exploring implementation of policies, it is also important to specify the time period within which progress in policy implementation is assessed.
The geographical scope of the assessment should also be set before the assessment can begin. This is particularly important when assessing framework policies like EU policies that are implemented in specific national and sub-national contexts.
Finally, a key output of the structuring phase are the evaluation questions that will be addressed by the assessment. It is important to be specific about what you want to find out in order to guide the assessment. Defining sub-questions can help with this.
Examples
Examples of evaluation questions:
- To what extent are EU policies coherent with the selected European Green Deal goals and targets?
- To what extent are selected EU policies coherent with each other?
- To what extent are national-level policies coherent with EU policies and/or Green Deal targets?
Examples of sub-questions:
- Are the policy objectives aligned substantively, geographically, and temporally?
- How do the measures of policy A support the objectives and measures of policy B, and vice versa?
Phase 2: Data collection
In the second phase, the data for the assessment is collected. The type of data needed, and appropriate data collection methods will depend on the evaluation question(s) and on whether you intend to conduct a light or in-depth assessment. (The two assessment types are explained in chapter 3.) Typically, light assessments can be conducted based on expert or stakeholder opinion. This can provide a first mapping before an in-depth assessment, or be the only assessment. For the in-depth coherence assessments, desk-based studies that involve the analysis of policy documents, legal acts, secondary literature is recommended, in combination with expert interviews.
The first step in Phase 2 is to determine what kind of data and information are needed to answer the evaluation question(s). Basic data needed for a coherence assessment are the objectives, targets and measures of the assessed policies. Policy objectives are defined as the results and outcomes the policy sets out to achieve, as specified in the articles or text of the policy document. Measures are the specific actions that need to be taken to deliver the policy objectives. It is important to understand their alignment on paper as well as how these objectives and measures interact during decision-making, policymaking, and planning processes. For a more comprehensive description of policy objectives and measures, and examples, see (hyperlink).
The second step consists of determining the relevant data sources and methods to collect this data. The following box provides an overview of possible data sources and collection methods for the assessment of coherence.
Sources and methods for collecting data
1. Text analysis of legislative and policy documents
Data is collected from the actual policy document text, as well as related documents and case law. Related documents can include evaluation reports and impact assessments, guidelines, explanatory memorandums and other documents that aim to explain the intention or anticipated effects of the policy.
2. Analysis of academic and policy reports
Data is collected from policy reports and research studies. This includes impact assessments, fitness checks, strategic environmental assessments, mid-term and ex-post evaluations studies. It also includes academic literature on scholarly analyses of specific policies.
3. Surveys (interviews and questionnaires)
Data is collected through interviews or questionnaires. Survey participants can be experts, officials, stakeholders or anyone who has relevant knowledge of, or a concrete interest in, the subject under investigation.
4. Stakeholder workshops
Data is collected from a group of stakeholders or experts through workshops, expert panels or focus groups.
5. Case studies
Data is collected from case studies. Case studies can be an important approach to better understand the causal pathways between policy design, implementation and impacts. It is possible to study single cases, though more insight can be gained by comparing findings across different case studies.
Phase 3: Data analysis
Phase 3 is the main assessment phase. In this phase, the collected data will be analysed to address the evaluation questions (hyperlink) defined in Phase 1. The CrossGov Policy Coherence Framework can be used for the data analysis phase.
Note that the guidance from this framework can be used when conducting policy assessments and evaluations according to EU’s Better Regulation guidelines. See especially chapter 5 of this handbook for a description
Phase 4: Synthesis and conclusions
In the last phase, the results of the analysis are pulled together to provide answers to the main evaluation questions and to draw overall conclusions to assessment’s overall research question. Aggregated scoring approaches may be useful for presentations, policy briefs, or publications. Apart from that, it is recommended to maintain detailed assessments of the various policies and their attributes as these provide valuable insight into specific challenges and obstacles towards policy coherence.