Since the 2000 pilot report, seven full versions of the European Innovation Scoreboard have been published. The list and number of indicators has undergone major changes over time as highlighted in Table 5. The number of indicators has increased from 18 to 25 and those derived from the Community Innovation Survey from 4 to 7 [1]. With major revisions in 2003 and 2005 (the dissimilarity percentages exceed 30 in both years), only 13 indicators feature in all Scoreboards. The number of countries has increased to 37, although actual data availability varies from very good (90% or more) for most EU27 countries, Norway and Switzerland, to good for Bulgaria, Cyprus, Latvia, Slovenia, UK and Iceland (between 75% and 90%), to moderate for US, Israel and Australia (between 60% and 70%) and to poor for Croatia, Turkey, Japan and Canada (less than 60%). The EIS indicators are grouped in different categories to capture key dimensions of the innovation process. In 2005 the current five dimensions were introduced. Overall innovation performance is captured by a composite index, the Summary Innovation Index, which has also been revised several times, most recently in 2005 following the EIS 2005 Methodology Report.
Current and past versions of the EIS and accompanying thematic papers have continuously tried to improve measurement of innovation performance by countries, sectors and regions. Future editions of the EIS will have to deal with a number of existing and new challenges under the following four headings:

• Measuring new forms of innovation
• Assessing overall innovation performance
• Improving comparability at  national, international and regional levels
• Measuring progress and changes over time

Across these areas, there is a need to maximise the relevance and utility of the EIS for policy makers, programme managers, and the wider innovation community.

Measuring new forms of innovation
The changes in indicators and definitions of indicators used in the different EIS reports all reflect changes in our perception and understanding of the innovation process. [2] Innovation is a complex phenomenon where firms can use different models of innovation. Science-based innovation has been used by certain industries and large firms for a long time. Innovation and technological progress is here driven by firms by their new scientific discoveries. Innovation surveys were at first designed to measure science-based or R&D-based innovation. But new concepts of the innovation process have emerged. The model of user innovation, which was introduced in the 1980s, states that consumers and end users develop innovations. More recently the model of open innovation has emerged: companies can no longer rely on their own research but must instead combine own ideas and research with external research e.g. by buying licenses and other external knowledge. Many of the current EIS indicators are better suited to capture science-based innovation. Therefore, new indicators are increasingly required to better capture new trends in innovation as portrayed in the models of user and in particular open innovation, for example on measuring knowledge flows.

Services innovation is becoming more and more important as the relative size of the services sector in the economy is continuously increasing. Innovation in services may differ from that in manufacturing e.g. by greater use of marketing and organisational innovation. Also service innovations may be increasingly prevalent within manufacturing sectors. Current statistics and innovation policies are biased towards measuring technological innovation and with therefore new developments in both statistics and policies may be needed for better understanding and stimulating non-technological innovation.

To improve the measurement of new forms of innovation in future editions of the EIS we need to develop and implement new indicators measuring e.g. open innovation, user innovation and non-R&D innovation. New indicators can draw on new data, in particular the improved measurement on marketing and organisational innovation and services innovation in the latest editions of the Community Innovation Survey, but more improvements are needed to fully capture all innovation process in the European economies.

Assessing overall innovation performance
The EIS provides a composite index, the Summary Innovation Index, which summarises innovation performance by aggregating the various indicators for each country in one single number. The 2005 Methodology Report studied in detail alternative computation schemes for the SII, but recent developments in composite indicator theory may call for changes in the scheme. The SII transforms each indicator on a relative basis, i.e. each indicator is measured relative to the best and worst performing country. Some of the indicators are highly skewed, e.g. patent applications. The question emerges whether or not to transform the indicators as for many of the indicators the distribution of the data differ from the normal distribution on which composite indicator theory is based.

In addition, the EIS provides innovation performance by 5 groups of indicators, the innovation dimensions. This helps to capture the overall innovation environment in a country. But with the innovation process becoming more complex, new innovation dimensions may emerge which should be included in the EIS. The current EIS distinguishes between input and output indicators, with about 50% more indicators measuring innovation inputs then outputs. This is due to the greater number and maturity of many input indicators, such as R&D expenditures. But just as companies are more interested in their profits or the final results of their production activities, should the EIS not focus more in the future on measuring the outputs of the innovation process? And is it justified to classify the indicators in input and output indicators only or should be also introduce process or throughput indicators? In particular for the patent indicators it is questionable if these are true output indicators instead of input or process indicators.

Assessing innovation performance inherently also covers assessing the efficiency of the innovation process. [3] Countries can increase their innovation performance by improving the efficiency of their innovation process without having to increase their innovation inputs. It is essential to continue to improve the measurement of the level of innovation efficiency correctly and to identify areas of improvement, drawing on academic studies in this area? [4]

Countries also differ in their state of economic development, in their industrial specialisation patterns and in their need for innovation driving their current and future well-being. Clearly not all countries have to invest as heavily in innovation as some of the innovation leaders do; other strategies for improving economic well-being are more realistic for those countries relying on productivity improvements driven by increases in other production factors. How could differences in the industrial structure between countries be taken better into account when benchmarking their innovation performance? Should different measures of innovation performance be applied depending on the type and/ or level of innovative activity in a country?

Should the EIS include wider socio-economic factors? For example governance and market indicators could provide useful information for policy makers about the environment for innovation. Innovation as such is not a goal in itself, companies innovate to improve their performance and countries similarly innovate to improve their economic performance. Should the EIS include economic indicators as a second layer of output or outcome indicators to measure the effect of innovation on the economic performance of a country?

Improving comparability at national, international and regional levels
Comparability issues arise within the EU due to differences between Member States in methodologies or sampling methods for collecting their data. Some of the EIS indicators are subject to national contexts (e.g. what constitutes tertiary education) which makes cross country comparisons difficult. In particular comparability difficulties arise in the Community Innovation Survey, where differences in the perception of innovativeness (e.g. the perception the sales share of new-to-market products) between countries may hamper the comparability of the results between the Member States. Further improvements are needed to ensure that differences in people’s and firms’ perception across Europe do not bias the comparisons of innovation performance.

In a globalising world, the EU needs to compare itself with emerging competitors and the EIS therefore may need to include more non-EU countries. For ensuring comparable benchmark results, data should be collected from harmonized databases supplied by international institutes as the OECD or the World Bank. There is also a need to eliminate biases between the EU and other regions in IP data, with EU Member States experiencing home advantages in EPO patents, Community trademarks and Community designs and the US in USPTO patents. Other comparability problems arise from the non-existence of innovation surveys in many non-EU countries or differences in the survey questions or methodologies between the EU countries and non-EU countries. How should the globalising EIS deal with these issues? Should it aim at including as many indicators as possible or select a core set of indicators for which data are available for all countries? [5]

At present, innovation at the regional level is captures in the Regional Innovation Scoreboard (RIS) [6] which attempts to use the same methodology as the EIS, but with significantly reduced data availability. The RIS is seriously hampered by the non-availability of regional CIS data and regional data for many of the other indicators. Data are not available as these are either not collected as such the national statistical offices (NSO) or they are considered to be unreliable due to sampling methods . Another problem arises from the location of the headquarters of a company and where the regional activities of a company are reported, at the respective region or at the headquarters’ region? What could be done to improve data availability and its accuracy in assigning inputs and outputs to the correct geographical region?

Measuring progress and changes over time
The EIS is currently designed as a tool for comparing innovation performance across Member States and other countries. In the past there have also been country specific assessments. However, changes in innovation performance over time also need to be measured to allow countries and regions to monitor progress in their innovation performance and to analyse the impacts of innovation policies on aggregate performance. At the EU level, better measurement of changes in innovation performance over time could be used to further assess progress against national reform programmes under the Lisbon strategy, and to underpin the Open Method of Coordination approach whereby countries benchmark their performance and set voluntarily targets.

All of this requires a sound and robust measurement of innovation performance over time. The current EIS is constructed as a measure of relative changes in innovation performance vis-à-vis other countries in the sample, where, due to the observed general process of convergence, the best performing countries show a relative decline in their SII scores and the worst performing countries an increase in their SII scores. The overall policy relevance of the EIS could improve if it also allowed to measure improvements in absolute innovation performance, creating opportunities for policy makers to use the EIS as a tool to set objectives, monitor performance and evaluate past policies so as to improve future innovation policies. In addition, there is currently a constraint in using the EIS to monitor progress due to the delays of several years in the availability of many indicators. Therefore ways should be explored to improve the timeliness of the indicators such that policy makers have more up to date measurements of performance.

Measuring the dynamics of innovation performance over time may also require new approaches, such as considering trends over longer time periods, whether time lags should be introduced for some input indicators, and whether it would be appropriate to model stocks of innovative capabilities that accumulate over time.