Methodology report

The EIS 2008 Methodology Report explains in detail the new methodology that has been used for the EIS 2008 report and also intended for the 2009 and 2010 reports. The revision of the EIS methodology was a direct result of the challenges discussed in the EIS 2007 report to: 1) measure new forms of innovation; 2) assess overall innovation performance; 3) improve comparability at national, regional and international levels; and 4) measure progress and changes over time.

Over the years the EIS has received a number of criticisms such as the lack of an underlying rationale for the choice of innovation dimensions and indicators; for using composite indicators and ranking tables; for being biased to measuring innovation in high-tech industries; for the fact that several of its indicators are highly correlated; and for the underlying assumption that a higher score on an indicator implies a better innovation performance (a review of published criticisms of the EIS is provided in the 2008 methodology report).

 

Figure 13: EIS Revision process

The revised methodology has not only tried to address the above challenges and criticisms, but the revision process has also actively involved the participation of many stakeholders, from academic researchers to policy makers and Member States’ representatives (cf. Figure 13). Stakeholders were invited to participate in the June 16 EIS workshop “Improving the European Innovation Scoreboard methodology” in Brussels, discussing in detail the challenges for measuring innovation performance. The workshop input report prepared by UNU-MERIT presented a first draft of a revised list of innovation dimensions and indicators and report prepared by the Joint Research Centre (JRC) discussed a range of different composite indicator growth formulas measuring real progress over time. The workshop’s discussions on dimensions and indicators resulted in a revised output report discussing an updated draft of a new set of innovation dimensions and indicators [1]. Further work on the feasibility of adopting the new dimensions and indicators and more discussions with some of the stakeholders has resulted in the final list of indicators as shown in Table 1.

During the revision process three principles were applied in considering possibilities for improvement: 1) Simplicity such that the number of indicators is limited as compared to other studies and will not undergo unnecessary manipulations; 2) Transparency such that all results can be easily recalculated, based on a careful and detailed explanation of the methodology for calculating the composite innovation indicators; and 3) a reasonable level of continuity with previous and future years such that the results between the new EIS 2008 will be directly comparable to those of the EIS 2009 and EIS 2010 and the results of the EIS 2000-2007.

The revised methodology is presented in the Methodology Report published in September 2008 and it presents a short rationale for including each indicator and concise definitions.

The new methodology also includes a revised method of calculating countries’ average innovation performance allowing tracking the development of individual innovation performance over time. As with any benchmarking exercise, the inherent assumption is that innovation performance can be measured using the same set of indicators despite the fact that there are differences in countries’ innovation systems. The new methodology only uses internationally comparable statistics that are regularly updated, and is therefore limited by the availability and timeliness of such data. It is intended to maintain the same methodology for the 2009 and 2010 editions of the European Innovation Scoreboard to allow direct comparability between reports, while at the same time exploring the potential of new statistical sources through the EIS thematic reports.

 

---