(p.228) Appendix 3 System 2. Indicators for internal evaluation
(p.228) Appendix 3 System 2. Indicators for internal evaluation
This system contains two different sets of indicators: a checklist and indicators of real-world outcome.
The objective of the checklist is to provide accurate monitoring of major regulations and activity in the areas of consultation, simplification and access. The checklist should be compiled every year by governments and the Commission in order to show how specific issues were taken care of and to show progress over time. Essentially, it arises out of the previous set of indicators on ‘quality of the process’, specifically on the following set of indicators:
• RIA guidance;
• consultation guidance;
• elements of simplification strategy;
• components of the access strategy.
2.A Regulatory impact assessment
For each preliminary RIA or road map
[For governments making use of the two-stage RIA]
1 Does the preliminary RIA define the problem in terms of its magnitude, causes and the incentives of the various parties?
2 Does it make use of explicit criteria to select the proposals deserving extended RIA?
3 Does it perform the subsidiarity test (whether regulation, if needed at all, is more efficient at the EU, national, or sub-national level)?
4 Does it check the compatibility of proposed national rules with the single-market legislation and the acquis?
(p.229) For each (final – in the case of a two-stage approach) RIA report
1 Make use of criteria to select (when applicable, on the basis of preliminary RIA or road maps) the proposals deserving RIA?
2 Use the maximisation of collective welfare as the main criterion of choice?
3 Define the problem in terms of its magnitude, causes and the incentives of the various parties?
4 Identify the baseline situation and its probable change over time sufficiently clearly to compare it with proposed changes?
5 Articulate policy objectives as measurable outcomes, goals or targets?
6 Justify government or Commission action considering the option of no action or of market solutions?
7 Identify all parties that are substantially affected by the change?
8 Estimate the lifetime of the policy or options?
9 Identify the negative impacts on all those affected for each option?
10 Include a risk–risk analysis comparing the risks of the present situation with those of possible solutions?
11 Provide evidence that consultation informed the assessment of different options?
12 Contain evidence of consultation with other governments and information on checks on overlapping regulations (for regulations likely to impact on other levels of governance)?
13 Express uncertainty about the future in terms of scenarios or probability?
14 Provide evidence of:
1 an explicit set of values
2 a framework, or
3 a model
to identify and assess the trade-offs (in terms of the Lisbon goals of growth, social cohesion and environmental protection) between alternative policy options? Discuss the distribution of the positive and negative impacts?
15 Assess different options with reference to how they will perform in terms of enforcement and compliance levels?
16 Analyse options in terms of their impacts on market structure and competition?
17 Include simulation, pilots and ‘dry runs’ of new forms (for regulations introducing new forms for citizens and firms)?
18 Provide – when relevant – a discussion of the distribution of the negative and positive impacts (among different groups of people and/or different geographical areas)?
19 Justify why no review clause is proposed (the default case being that there is a review clause in proposed regulation1)?
20 Contain procedures for monitoring and evaluating ex-post the extent to which the regulation meets its objectives 21 Contain evidence of external peer review (for the most important regulations)?
22 Include a contact name and website where all underlying data used in RIA are available?
(p.230) 2.B Consultation
For each consultation of major regulations
Do the consultation reports (or consultation sections of an RIA) contain the following?
1 Evidence of compliance with the timetable for consultation
2 A consultation timetable and planning which enabled those affected to comment on the final assessment
3 Evidence of compliance with standards on how to deal with experts' advice at the stage of policy formulation
4 Justification of the methodology chosen for consultation
5 Evidence of use of innovative consultation methods – when potential participants have difficulty in participating in traditional ways
6 Evidence in consultation or RIA documentation that affected parties were involved in the discussion of the problem and the identification of the possible alternative solutions
7 Evidence in consultation or RIA documentation of the use of a procedure allowing the general public to participate in the consultation process
8 Evidence in consultation or RIA documentation of compliance with the principle that consultation goes beyond a one-off event
9 Presence in consultation or RIA documentation of supporting documents explaining the impact of different alternatives
10 Evidence of how the input provided by consultation was considered and responded to
11 A summary of the results of consultation
Has simplification activity in a given year delivered the following?
1 Meeting the annual timetable deadlines
2 Information on how quantifiable targets were met
3 Information on how change and the results of simplification were measured
4 Review of the simplification programme (at least every three years) in the light of the results achieved
5 Evidence that the introduction of major new legislation has been accompanied by the simplification of existing regulations
6 Use RIA for the simplification of major rules affecting business and/or citizens
7 Evidence that simplification of major regulations is consistent with the single market and WTO
8 Use of review or sunset clauses when new rules meet the following conditions:
• Regulation was introduced at short notice
• Regulation is based on the precautionary principle
• Technology or market conditions are most likely to change
• Regulation is a pilot project
• Regulation confers rights upon the state
9 Public register of regulatory provisions is accessible on-line
(p.231) 2.D Access
Does the government's or Commission's activity on access in a given year include the following elements:
1 Code of plain-language drafting
2 Annual regulatory agendas available on-line
3 Evidence on the systematic use of procedures to enable users to complain about the difficulties in complying with regulation
4 Evidence that the right of citizens to access the documents underpinning a rule is exploited
5 Documents available on-line through a single access portal. For example:
• Documents supporting consultations
• Summary of consultation results
• Comments from interested parties
• Responses to comments
• All texts recalled in and supporting RIA
Real-world outcome indicators
Turning to the set of indicators of real-world outcome, their purpose is to evaluate the quality of better regulation policies by looking at their impact. We suggest three classes of real-world indicators:
2.E Indicators of the impact of better regulation policies
These are indicators tracking down the impact of better regulation policies and tools in critical areas. One has to be careful to distinguish real-world indicators that measure the impact of regulatory policies from real-world indicators that measure the impact of better regulation policies. An indicator like ‘Percentage of annual deaths in industrial accidents avoided by new regulations’ measures the impact of the former. However, our study is focused on the latter dimension. The key issue is whether the cost of producing a unit of benefits is declining. Accordingly, our indicators focus on this issue.
A classic, although limited, indicator is the reduction of regulatory costs. Within regulatory costs, there is a political determination in the EU to target a specific subset, that is, administrative burdens. The EU has made reference to the reduction of administrative burdens as a possible indicator. We have reservations about using this as either the only or the most important measure of quality. However, in the context of a broader set of measures, this could be a valuable indicator. Moreover, the fact that there is consensus in the EU on this measure makes it an obvious reference point.
1 [Estimate of total regulatory costs delivered by regulations for which RIAs were prepared in year xxxx] divided by [Estimate of total regulatory benefits delivered by regulations for which RIAs were prepared in year xxxx]
2 [Net benefits delivered by regulations for which RIAs were prepared in year t] divided by [Net benefits delivered by regulations for which RIAs were prepared in year t – 1]
(p.232) 3 Total cost reduction resulting from simplification in year xxxx
4 Annual rate of reduction in the total administrative burdens stemming from EU legislation (assuming a quantifiable target to reduce the total administrative burdens by 25 per cent in five years – an objective pursued in some countries such as the Netherlands and Belgium – the value of this rate is 5 per cent)
5 [Cost of administrative procedures eliminated in year t] divided by [Cost of administrative procedures eliminated in year t – 1]
2.F Survey-based indicators
As illustrated throughout the book (chapter 2 in particular), one important set of measures of impact should target changes in the behaviour of regulators and key stakeholders. Although it is impossible to measure behavioural change at reasonable cost, surveys of attitudes and perceptions show changes in thinking and culture that inform behaviour. We discussed the limitations of surveys in chapter 3, but we also highlighted their strengths. Consequently, we propose a limited set of survey measures.
We propose a set of indicators which have either been already gathered ad hoc, for special Eurobarometers, or which can be gathered by the EBTP.2 These indicators should be used in a trend perspective, that is, across time. It is more important to measure progress (or lack of it) across time than to establish ex-ante thresholds.
1 Percentage of citizens who think that their views are taken on board in the development of single-market and enterprise policy in the EU.3
2 Percentage of firms which think that the quality of regulation in the EU (including both EU and domestic rules) has improved over the last three years, making it easier to do business in Europe.4
3 Percentage of firms which think that regulators consult the business sector appropriately before taking decisions on new regulations.
4 Percentage of firms which think that participation in EU-level consultation makes an impact on the final decision
5 Percentage of firms which say that they have participated at least once over the last five years in EU-level consultation
6 Percentage of firms which think that the EU better regulation policy has an impact in opening markets, in making it easier to compete and in generating competition in the manufacturing sector
7 Percentage of firms which think that the EU better regulation policy has an impact in opening markets, in making it easier to compete and in generating competition in the service sector
8 Percentage of firms which think that the EU better regulation policy has an impact in opening markets, in making it easier to compete and in generating competition in the professions
9 Percentage of firms which think that in last three years restrictions and obstacles to their business have disappeared altogether or been significantly reduced
10 Percentage of firms which think that regulatory changes at the EU level are predictable
11 Percentage of firms which think that access to information on the single market has improved over the last three years or so
(p.233) Survey of regulators
There is no systematic survey of regulators at the moment. We would suggest that the Commission consider a biannual survey. We do not prescibe particular contents of a survey. However, to illustrate with an example, we tentatively suggest the following survey questions:
Percentage of regulators who think that RIA has the following impact:
1 Makes policy formulation more transparent
2 Speeds up/delays policy implementation
3 Reduces conflict at the stage of policy formulation
4 Reduces conflict at the stage of policy implementation
5 Is more likely to represent the preferences of affected citizens
6 Increases the likelihood that the final policy decision will be lower cost
7 Increases the likelihood that the final policy decision will be more likely to achieve policy goals
8 more likely to generate a consensus among affected entities
9 more likely to be based on economic analysis
10 more likely to check one risk against other possible risks
11 more responsive to technological innovation
12 more likely to create open markets
13 more likely to create fewer trade controversies in transatlantic relations
2.G Indicators of economic outcomes
These are discussed in detail in chapter 3. In the context of this study, indicators of economic outcomes are very useful, but caveats should be considered. They are an important conceptual link between better regulation policies and the Lisbon goals. The critical issue is whether indicators of economic outcome are correlated with better regulation policies – controlling for spurious correlation. In our view, it is eminently a matter of long versus short causal chains. Some indicators (e.g., employment levels) can be linked to better regulation policies only via long causal chains, for example by arguing that ‘better regulation’ policies create the preconditions for better regulatory policies, that better rules produce a better business environment and flexible labour markets, that firms respond to the environment by investing, and that investment in the long term produces more jobs. At each step of this long causal chain one has to add ceteris paribus. A high number of ceteris paribus assumptions make the long causal chain unrealistic. In contrast, the chain between better regulation policies and the flexibility of the internal market is shorter and more realistic.
We distinguish between composite measures and simple measures. In the case of composite indicators, we specify the causal chain going from better regulation policies to the indicator. The indicators we propose should be measured across time to evaluate progress (or lack of it) and whether better regulation policies are contributing effectively to the Lisbon agenda. Hence we propose a consideration of the rate of change of the following indicators.
(p.234) Composite indicators
 Internal market index
Causal chain: Better regulation policy produces better regulatory policies. This should be reflected in the decreasing importance of economic interventionism (state aid should decrease) and in open markets for the utilities. Consequently, the price of gas, electricity and telecommunications should decrease. Investment in an open and flexible internal market should increase. The internal market index is a composite measure of these variables.
Description: ‘The Internal Market Index is intended to track the progress of the EU Internal Market towards becoming a fully functioning market’ (European Commission, 2004c: 20). The index is composed of twelve ‘individual indicators, which are all considered relevant to the Internal Market's development’, as set out below; an indication is also given of the relative importance of each indicator (European Commission, 2004c: 20). It is presented each year on the Internal Market Scorecard, the publication edited by DG Internal Market. The individual indicators are the following (Tarantola et al., 2002: 25):
1 sectoral and ad hoc state aid (as a percentage of GDP)
2 value of published public procurement (as a percentage of GDP)
3 telecommunications costs (in euros)
4 electricity prices (in euros)
5 gas prices (in euros)
6 relative price level of private final consumption, including indirect taxes
7 intra-EU foreign direct investment inward flows (as a percentage of GDP)
8 intra-EU trade (as a percentage of GDP)
9 active population in a Member State (aged 15–64) originally coming from another Member State (per capita)
10 value of pension fund assets (as a percentage of GDP)
11 retail lending interest rates over savings interest rates ratio
12 postal tariffs (20 g standard letter) (in euros)
The aggregation method is twofold. Firstly, there is a preliminary treatment of data in order to get the percentage year-to-year differences for each indicator. The resulting data are processed using the statistical method of principal components analysis.5 This is a useful method ‘to weigh the influence of each variable on the final score’ (Joint Research Centre of the European Commission, 2002: 25).6
 Indicators of product market regulation
Causal chain: Better regulation policy delivers better regulatory policies. In consequence, the overall extent of regulation should diminish as a result of the systematic use of RIA, simplification, consultation and access. Smarter institutions can see the benefits of using market-friendly alternatives to regulation.
Description: Indicators of product market regulation have been developed by the OECD. They are discussed in chapter 3. These indicators rely on a detailed database covering both economic and administrative types of regulation and measures the extent of regulation. The database, set up in 1998, has been updated. A new dataset (p.235) was released in April 2005 (see chapter 3). The index facilitates the evaluation of regulatory tools and can be applied to specific policies (different but related to better regulation policy, e.g. innovation policy).
 Summary innovation index (innovation scorecard)
Causal chain: Better regulation policy produces better regulatory policies. In turn, good regulation stimulates innovation.
Description: This index is produced by the Commission (DG Research).7 It builds on the ‘structural indicators’ but with the aim of focusing on innovation policy. The Summary Innovation Index (SII) for a given country is equal to the number of indicators that are more than 20 per cent above the EU overall mean, minus the number of indicators that are more than 20 per cent below. The SII is adjusted for differences in the number of available indicators for each country; the index can vary between +10 (all indicators are above average) to –10 (all indicators are below average).
 Performance in the knowledge-based economy
Causal chain: Better regulation policy produces better regulatory policies. In turn, good regulations stimulate gross fixed capital formation, investment in new sectors like e-commerce and overall productivity. These variables are included in these two complex measures. However, there are other variables for which the causal linkage is not strong. Specifically, the causal chain is weaker in relation to variables like spending on education or number of researchers.
Description: Both ‘indicators attempt to capture the complex, multidimensional nature of the knowledge-based economy by aggregating a number of key variables, and expressing the result in the form of an overall index’ (European Commission, 2003g). In other words, they measure Europe's performance in research and innovation.
DG Research and the Joint Research Centre collect both indexes.
Index  is composed of the following seven sub-indicators:
• domestic expenditure on research and development (per capita);
• number of PhDs (number of new science and technology PhDs per capita);
• number of researchers (number of researchers per capita);
• gross fixed capital formation (excluding buildings) (per capita),
• the share of e–government;
• educational spending;
• lifelong learning.
(p.236) Index  is composed of five sub-indicators:
• overall productivity (GDP per hour worked);
• patents (share of EPO and USPTO patents);
• scientific publications per capita;
• schooling success rate.
The indexes are based on the same aggregation technique – a three-stage methodology. The first step is to calculate the mean and the dispersion index (standard deviation) for each indicator. The second step consists of obtaining ‘country by country and year by year the original value for each indicator’, which ‘are converted by “centring” them on this mean and dividing them by this dispersion index’. Finally, ‘the value of the composite indicator for each country and for each year is the weighted average of the values of all the indicators’ (Joint Research Centre of the European Commission, 2002: 27).
Both indicators are contained in the DG Research publication Key Figures on science, technology and innovation.8 Interestingly, this publication benchmarks the indexes for Europe with the USA and Japan. This is relevant in connection to the Lisbon agenda to make the EU the most competitive knowledge-based society in the world.
Thus,  and  could also be measured in the following way:
4 – alternative measurement. [Rate of change of: knowledge-based investment (EU)] divided by [knowledge-based investment (USA)]
5 – alternative measurement. [Rate of change of: knowledge-based performance (EU)] divided by [knowledge-based performance (USA)]
To sum up,  and  have a relatively robust causal chain. ,  and  provide useful information, but the causal chain with better regulation is weaker, especially for  and . Accordingly, they should not be seen as key indicators to gauge the success or failure of better regulation policies. Instead, ,  and  should be considered important background indicators. They provide a direct link with the Lisbon strategy, although their value is only in minor part determined by better regulation policies.
(p.237) Single measures
Single structural indicators can be used track the progress made in the Lisbon strategy. They are key measures of real-world outcome, although it is impossible to establish a clear causal relationship between each of them and better regulation policies. We suggest they are considered across time and with reference to US performance.
Annual rate of change of the following indicators:
1 Industry-level multifactor productivity growth (collected by the OECD)
2 Variation in the net growth of start-ups (new firms minus firms that have ceased activity) (collected by Eurostat)
3 Rate of investment in high-growth sectors (telecommunications and bio-technology)
4 Time needed to set up a new firm (collected by World Bank, but not every year)
5 Administrative costs to start up a new firm (collected by World Bank, but not every year)
6 Number of procedures to hire (to fire) the first (nth) employee (collected by World Bank, but not every year)
7 Time needed to hire (to fire) the first (nth) employee (collected by World Bank, but not every year)
8 Administrative costs to hire (to fire) the first (nth) employee (collected by World Bank, but not every year)
(3) This question draws and expands on previous experience with Eurobarometer. In October 2001 Eurobarometer reported on the percentage of citizens who would like to take part in a ‘dialogue on Europe’: 26 per cent said that they would, but 62 per cent said that they would not; 29 per cent agreed with the statement that ‘my view would not be taken into account anyway’ and 17 per cent agreed that ‘it would be a waste of time’.
(4) The reason for not making a distinction between EU rules and domestic rules is that more often than not the firm cannot distinguish between EU rules, the domestic implementation of EU rules and domestic rules.
(5) This is a statistical methodology often used in the construction of complex indicators.