Abstract
Paul Cunningham
Abdullah Gök
Manchester Institute of Innovation Research, University of Manchester
Manchester Institute of Innovation Research, University of Manchester
Nesta Working Paper 12/06
March 2012
www.nesta.org.uk/wp12-06
This paper is part of the Compendium of Evidence on the Effectiveness of Innovation Policy Intervention. It examines the evidence on the effectiveness of publicly supported schemes that aim to promote or enhance collaborative innovation activities between firms and, what may broadly be termed, the science base - i.e. public laboratories and research institutes and Higher Education Institutions, particularly universities. It bases its analysis on number of evaluation reports and the academic literature. The evidence reviewed is organised around the issues of input additionality, output additionality, behavioural additionality and programme design and governance. It also discusses five important thematic issues including the evaluation of the Alvey Programme, evaluations of the EUREKA Programme, Japanese experience with collaborative R&D support, CIS based evaluations of collaboration, and finally, a series of evaluations of Australia's Cooperative Research Centres Programmes. The report proposes a set of general lessons for the design and implementation of collaborative support instruments, i.e. the typical pre-conditions for success. Programme success is also found to be closely aligned with the characteristics of the participants. The report also has important conclusions for programme governance while a final set of conclusions relates to lessons for evaluation and future research needs.
JEL Classification: 038
Keywords: Impact, Innovation policies, R&D collaboration schemes
The Compendium of Evidence on the Effectiveness of Innovation Policy Intervention Project is led by the Manchester Institute of Innovation Research (MIoIR), University of Manchester, and funded by Nesta, an independent charity with the mission to make the UK more innovative. The compendium is organised around 20 innovation policy topics categorised primarily according to their policy objectives. Currently, some of these reports are available. All reports are available at http://www.innovation-policy.org.uk. Also at this location is an online strategic intelligence tool with an extensive list of references that present evidence for the effectiveness of each particular innovation policy objective. Summaries and download links are provided for key references. These can also be reached by clicking in the references in this document. Corresponding Author: Dr Paul Cunningham, MIOIR, Harold Hankins Building, Manchester Business School, University of Manchester, M13 9PL, [email protected]
The Nesta Working Paper Series is intended to make available early results of research undertaken or supported by Nesta and its partners in order to elicit comments and suggestions for revisions and to encourage discussion and further debate prior to publication (ISSN 2050-9820). © 2012 by the author(s). Short sections of text, tables and figures may be reproduced without explicit permission provided that full credit is given to the source. The views expressed in this working paper are those of the author(s) and do not necessarily represent those of Nesta.
This report is part of the Compendium of Evidence on the Effectiveness of Innovation Policy Intervention Project led by the Manchester Institute of Innovation Research (MIoIR), University of Manchester. The project is funded by the National Endowment for Science, Technology and the Arts (NESTA) – an independent body with the mission to make the UK more innovative.
The compendium is organised around 20 innovation policy topics categorised primarily according to their policy objectives. Currently, some of these reports are available.
All reports are available at http://www.innovation-policy.org.uk. Also at this location is an online strategic intelligence tool with an extensive list of references that present evidence for the effectiveness of each particular innovation policy objective. Summaries and download links are provided for key references. These can also be reached by clicking in the references in this document.
Executive Summary
The benefits arising from research collaboration are clear, and have long been recognised by those working both in the public (academic) and private sectors. There is a comparatively long history regarding measures to foster longer-term cooperation between science and industrial actors and these now represent a significant part of the portfolio of innovation policy support measures in many countries. This report focuses on the evidence on the effectiveness of the publicly supported schemes that aim to promote or enhance collaborative innovation activities between firms and, what may broadly be termed, the science base – i.e. public laboratories and research institutes and Higher Education Institutions, particularly universities.
Firms collaborate with each other for a number of reasons: i) saving transaction costs where there are incomplete contracts, ii) attaining economies of scale and scope, iii) using networks as a way to increase synergy, efficiency and power, iv) accessing complementary resources to exploit firms' own resources and learning from partners, thereby creating new capabilities and thus enhancing competitiveness, v) creating and exploiting high risk high opportunity situations, vi) decreasing R&D costs by pooling risks and co-opting competition. They also collaborate with universities in order to access leading edge research knowledge, research infrastructures or research services, to develop in-house capabilities or to identify potential future employees. Universities collaborate with firms to access industrial capabilities and resources, to commercialise research ideas or test their commercial potential, to develop 'real world' links and build experience or to develop potential career pathways for students. Governments support these links due to economies of scope and scale, the need to internalise informational spillovers, the fact that collaborative projects can be too complex for single actors and the need for providing the medium for knowledge transfer.
The evaluations of collaborative schemes share several challenges common to the evaluation of other innovation support schemes. First, timing and periodicity pose an issue for evaluation of schemes for collaboration. While formal collaborative arrangements are comparatively easy to identify from their outset, several further anticipated outcomes and impacts are not likely to manifest until sometime into, or even after the completion of, the collaborative arrangement. Furthermore, collaborative arrangements develop and mature over time. Secondly, defining the scope of impact can be challenging. Although certain metrics may capture the overall outcomes of a programme, the outputs and experiences of individual projects and relationships may be difficult to discern. Third, it is difficult to identify the types of outcome and impact that arise from a collaborative relationship in the absence of counterfactual examples or benchmarks established prior to the formation of the collaborations. Finally, due to the increase of the significance of informal collaborative links, and the tacit nature of their outcomes and effects, the search for suitable metrics faces severe problems.
This report reviews the evidence on the effectiveness of innovation policy schemes for collaboration within a number of evaluation reports and the academic literature. The evidence reviewed is organised around the issues of input additionality, output additionality, behavioural additionality and programme design and governance. It also discusses five important thematic issues including the evaluation of the Alvey Programme, evaluations of the EUREKA Programme, Japanese experience with collaborative R&D support, CIS based evaluations of collaboration, and finally, a series of evaluations of Australia's Cooperative Research Centres Programmes.
In a final section this report proposes a set of general lessons for the design and implementation of collaborative support instruments, i.e. the typical pre-conditions for success. Regarding the programme design, it is concluded that some of the most successful measures are seen to be associated with a long-term, stable commitment of government funding and support. Our findings also highlight the importance of clearly defining the purpose of the programme in advance. It is also important to recognise that there are many reasons for which participants may wish to collaborate. For several reasons, it may become necessary to adjust the underlying rationale and objectives of a programme. Programme managers must be sensitive to these changes and be prepared to adjust the configuration of the programme and, at the same time, to reconfigure the ways in which performance is judged. Collaborative activities do not proceed in isolation. Thus, it is important to align collaborative support programmes within a broader supporting and complementary set of policies which can capitalise on their outcomes. Furthermore, the programme impact was found to be enhanced through the inclusion of an educational objective, such as the production of post-graduate students who were able to gain experience of project management in an industrial context.
As regards the selection of participants, programme success was found to be closely aligned with the characteristics of the participants. This implies a need for matching partners with a track record of collaboration with new partners or defining new areas and topics for collaboration.
Our review has important conclusions for programme governance. The management of collaboration initiatives is a major determinant of success, particularly in collaborations which involve novel partnerships, new research topics or where the anticipated research outcome cannot be guaranteed or the potential for unexpected outcomes is high. Furthermore, in long-term collaborative programmes it is important to define clear exit points and processes to allow for changes in partners and participants to be effectively managed whilst avoiding high turnover rates and the sudden loss of key partners. We also found that effective collaboration depends on mutual trust – that workloads and benefits will be shared equitably and that shared information will remain within the confines of the partnership. Moreover, strong governance and audit arrangements are closely associated with programme success. Perhaps most importantly, our review reveals that at all stages of the programme/project lifecycle, bureaucracy should be minimised. Finally, the fostering of a strong and positive brand image was found to increase the attractiveness of the scheme to high-quality participants, to increase motivation through a sense of 'belonging' and recognition and to increase the likelihood of additional networking throughout the programme rather than between the collaborating partners alone.
A final set of conclusions relates to lessons for evaluation and future research needs. The report concludes that evaluations must be sensitive to the complexity of the collaboration process, and the diversity of motivations, rationales, activities, outputs, outcomes and effects which it entails. Furthermore, the issue of timing is also prevalent in the evaluations of innovation policy measure for collaboration. We also found that the issue of causality is a particular methodological problem frequently encountered in the literature and evaluation reports.
1 Introduction
The benefits arising from research collaboration are clear, and have long been recognised by both those working in the public (academic) and private sectors. The benefits of bringing together groups of researchers and end-users are several and include: "the achievement of critical mass; overcoming fragmentation caused by distance and a smaller resource base; bringing together different perspectives, experience, skills and knowledge; breaking down specialist silos and restrictive organisational boundaries and fostering cross-disciplinary interactions; encouraging skills and knowledge transfer; promoting mutual understandings; and managing risks” (O'Kane, 2008). Although collaboration is intuitively more expected between firms or other entities that are not competitors, Cosh et al. (2005) find that “collaboration with customers, suppliers, higher education institutions, even competitors, allows firms to expand their range of expertise, develop specialist products, and achieve various other corporate objectives. Collaboration with competitors and customers provides a firm with greater access to domestic or international markets. This may lead to greater commercial success of the new products, and enhances the productivity of innovation through economics of scale. Collaboration with suppliers may lead to lower costs and better quality of the new products. All this may result in higher productivity of the innovation activities. Hence, collaboration will be positively associated with firms' innovative efficiency". This rationale provides a clear incentive for government support for the promotion of collaboration.
There is a comparatively long history regarding measures to foster longer-term cooperation between science and industrial actors and these now represent a significant part of the portfolio of innovation policy support measures in many countries (see OECD (2001), Tsipouri et al. (2006); Tsipouri et al. (2009)). For instance, as noted by Bruno et al. (2011), the US Engineering Research Centers programme has been in existence since 1985: during its lifetime it has played a major role in influencing the design of similar programmes in several other countries. During their evolution, there has been a shift in the primary rationale for collaborative support mechanisms, from what may be termed a set of 'technology transfer' objectives (based very much on the old linear model of innovation and which sought to directly transfer the results of public sector research into commercialisable products, process and services) towards 'knowledge transfer' objectives, which aim to optimise a broader range of innovation characteristics that are embodied in the systemic view of innovation and which involve less tangible interactions and feedback loops between the actors engaged. The latter objectives necessitate a more sophisticated policy design in order to optimise the full range of potential benefits arising from the collaboration. In turn, this poses a greater challenge for evaluating the success of such policy interventions since many of the outcomes and impacts are more subtle and less evident through simple metrics.
This report is part of a series produced under the NESTA Compendium of Evidence on the Effectiveness of Innovation Policy Intervention. The series aims to gather evidence of the impact of public intervention in support of innovation and to derive lessons for future policy. This report focuses on publicly supported schemes that aim to promote or enhance collaborative innovation activities between firms and, what may broadly be termed, the science base – i.e. public laboratories and research institutes and Higher Education Institutions, particularly universities.
2 Background
2.1 Conceptual framework
2.1.1 Collaboration - definitions for purpose of report
The academic literature presents a number of reasons as to why firms engage in research partnerships with other firms: these are summarised in Table 1. According to Hagedoorn et al. (2000:583), these reasons may include i) saving transaction costs where there are incomplete contracts, ii) attaining economies of scale and scope, iii) using networks as a way to increase synergy, efficiency and power, iv) accessing complementary resources to exploit firms' own resources and learning from partners, thereby creating new capabilities and thus enhancing competitiveness, v) creating and exploiting high risk high opportunity situations, vi) decreasing R&D costs by pooling risks and co-opting competition.
Several of the motives also apply to science-industry collaboration although the motives tend to be asymmetric: on the one hand, public sector actors may be seeking to access industrial capabilities and resources, to commercialise research ideas or test their commercial potential, to develop 'real world' links and build experience or to develop potential career pathways for students. On the other hand, firms may be seeking to access leading edge research knowledge, research infrastructures or research services, to develop in-house capabilities or to identify potential future employees, to name but a few.
Industry-academic links can be traced back as far as the late nineteenth century where they represented the main mechanism by which industry funded research, although their industrial and policy significance was not fully recognised until the 1970s, particularly in the US. The US was viewed as being a leader in the growth and development of industry-academic linkages and, by the 1980s, European countries, the UK in particular, had also adopted a range of supporting measures (Howells, et al 1998).
Four main mechanisms and sources underlie these linkages:
- informal contacts and spin-outs from university departments;
- contract and collaborative research performed by universities on behalf of industry;
- property-led initiatives in the form of science parks; and
- the commercial exploitation of university research through the management and licensing of intellectual property rights. (Howells, et al, 1998).
It is also frequently the case that the goal of increasing the level of science-industry interaction forms a subsidiary objective in very many innovation strategies. Thus, it is a very widespread form of support in many government policies.
Wilson (2012) cites research conducted by Imperial College which points to a number of reasons why university-business collaborations may not progress beyond the stage of initial discussions. In general terms, most of these could also be applied to the failure of business to business collaborations, especially numbers 7-9.:
- The needs of the business do not align with the mission and strategy of the university.
- Time scale and capacity mismatch; a university has already committed its resources and does not have the available capacity to meet the timescale that the business needs.
- Capability mismatch; a university does not have the skill set or the facilities to meet the needs of the business.
- The cycle of bureaucracy: where external funding is required, the bidding cycle does not meet the timescale the business needs.
- Financial constraints: a university is unable to provide the service required for the price the company is willing to pay. This is particularly apparent in the context of full economic costing in research collaboration where business input to the research merits valuation.
- Sustainability: the investment required by the university to provide the service does not have an acceptable payback period.
- Mismatch in expectations and objectives: expectations of outcomes from collaboration are not mutually recognised.
- Failure to agree on the future of the intellectual property that may be generated. Although much progress has been made in this area since the publication of the Lambert Intellectual Property agreements, it is still reported as a significant issue in some negotiations.
- Contrasting views on the management of indemnities and liabilities between prospective partners; viewed as being an increasing problem.
In section 2.1.3 below, we outline the main types of science-industry collaboration support schemes that may be found. In order to set the boundaries of our analysis, we broadly define such schemes as those aiming to stimulate or develop interaction between sources of research knowledge (such as public research laboratories and universities) and the potential users of such research (firms) which are centred on the joint undertaking of clearly identified research projects and which may also entail the placement of personnel (typically students) within an industrial location.
2 Taken from Hagedoorn et al. (2000:575)
2.1.2 Rationale for intervention
At a general level, collaboration is important to bridge the boundaries within the NIS and to allow the various actors to share their range of knowledge, skills, capabilities and competencies in order to promote innovation and drive competitiveness. According to O'Kane (2008), collaboration allows:
- Human and capital resources to be brought together with an ability to create an outcome that cannot effectively be done alone.
- Higher quality and more effective, integrated and robust outcomes, as each partner brings a differing perspective and experience to the process.
- Development of critical mass and overcoming fragmentation caused by factors such as distance, diverse jurisdictions and a smaller resource base.
- Government and government agencies to be partners not just facilitators (of particular relevance in promoting public good, not-for-profit research, and in solving social and environmental problems).
- The cross-fertilisation of ideas and mutual understandings and can help obtain commitment to decisions and outcomes.
- Linkages between research providers and research end-users, and encourage the transfer of skills and knowledge and the translation of new ideas into products and services.
- The sharing and management of risk in high-risk areas.
The rationales for policy intervention to promote science-industry collaboration are strongly supported by evidence from a range of studies. For instance, citing data from Australia, O'Kane (2008) finds that "businesses which engage in collaboration are 70% more likely to achieve new to the world novelty or creative innovation": however, those firm that collaborate are much more likely to do so with other firms rather than with research organisations, universities or governments. The 2005 Australian innovation survey found that 26% of innovating businesses were engaged in some form of collaboration. Out of these, only about 3% were engaged in collaboration with government organisations and about 2% were engaged in collaboration with Universities and other higher education institutes. Collaboration with overseas organisation was less than 1%.
Howells et al. (1998) chart the development of policies to promote HEI-industry linkages in the UK as a cumulative process in which new models were developed without necessarily displacing their predecessors: a situation of informal linkages and occasional bilateral contracts was supplemented by specific policy measures from the 1970s onwards. "These measures included the concept of ring-fencing areas of research deemed to be of particular relevance to industry, and promoting them with additional resources and management support. An early manifestation was the Directed Programme". Two schemes designed to support industrially relevant training also date from that time: Cooperative Awards in Science and Engineering (CASE) studentships and the Teaching Company Scheme [still in place and known as the Knowledge Transfer Partnerships scheme]. During the 1980s, beginning with the Alvey Programme (see thematic section) and Joint Opto-electronic Research Scheme, the mechanism of collaborative research between universities and industry was introduced, and continues to the present day in the form of the LINK programme and through participation by UK HEIs and industry in the European Union's (EU) Framework Programmes. The 1990s [saw] a broader effort to marshal academic research towards support for competitiveness and quality of life through the Foresight Programme and redefined missions for the Research Councils". Further schemes have followed and the UK, as with most other countries, now operates a number of schemes that specifically target cooperation, often in pursuit of allied policy goals.
Early work on industry-business collaboration in the UK identified access to research funding as the "prime motivating factor by higher education institutions (HEIs), but only as a means to pursue goals which fulfil the aims of both academic and industrial partners. Mutual trust and a professional, business-like approach by the academic partners are seen as the keys to success. Keeping the linkages over time is dependent upon good personal relationships and avoiding a divergence of objectives during projects" (Howells et al., 1998).
A recent review of university-business cooperation in the UK makes it clear that they operate within a rich ecology of interrelationships (often involving a diversity of intermediaries) and that the motives underlying such cooperation are diverse (Wilson, 2012). Within this ecology, a range of government interventions seek to facilitate interaction between the higher education and business sectors. The success of such interventions will be highly dependent on the ecology within which they operate. Thus, collaboration schemes are not just about knowledge exchange, but touch on a range of factors such as skills, employability, etc: "in complex interactive systems relatively minor changes to subsystems can, together, have a profound effect upon the performance of a system as a whole".
The primary aim of policy intervention is to bring about change in practices. Wilson (2012) notes that this can be stimulated by "various strategies: change motivated by good management, improving an organisation's performance in pursuit of its objectives; change motivated by direct or indirect funding incentives; change motivated by regulatory requirement". All three approaches have been used to stimulate business-university collaboration. However, the second of these is increasingly constrained by limitations on public spending, whilst the third, regulation, is often seen as burdensome and currently forms the target of moves to reduce its use. Therefore, Wilson's recommendations tend to focus on change motivated by leadership and good management (Wilson, 2012).
The Lambert (2003) Review can be regarded as a milestone in the development of business-universities relationships in the UK; since Lambert (2003) published his report, Wilson (2012) notes that there has been a huge expansion in collaborative activity in many dimensions, of mutual benefit to businesses, students and universities. Amongst the barriers to commercialisation of university IP, a lack of clarity over the ownership of IP in research collaborations was considered to be particularly important. A major outcome of the review was the framing of a set of guidelines which could be tailored to the needs of a range of collaborative arrangements established between universities and businesses. Whilst this was a significant policy intervention, which has certainly supported the practice of research collaboration between universities and industry, no formal evaluation of its effects has been identified .
Support for collaborative schemes is, generally, another form of direct support in that it frequently involves grant payments to the participating actors. It is predicated on the underlying rationale for the public support of R&D although there are a series of further rationales and advantages that are assumed to accrue from collaborative R&D undertakings. The primary rationale of such schemes is typically to improve the innovativeness of business communities and increase the social return from public investments in science.
However, there are a number of further rationales for such support which involve:
- Economies of scope
- Economies of scale
- The need to internalise informational spillovers
- Collaborative projects can be too complex for single actors
- Providing the medium for knowledge transfer
According to Hagedoorn et al. (2000), “Governments have promoted and supported research partnerships (between firms) in order to:
- correct market failures in R&D investment, particularly in the presence of highly non-appropriable research;
- speed up technological innovation, aiming at increased international competitiveness; and
- increase technological information exchange among firms, universities, public research institutes."
In its rationale for the Knowledge Transfer Partnerships Programme (formerly known as the Teaching Company Scheme) the UK Department for Business, Innovation and Skills (BIS) identified two justifications for the KTP programme, interestingly, based on market failures (Regeneris Consulting Ltd., 2010):
- Information failures - businesses often have difficulty in accessing or exploiting new knowledge to improve their competiveness.
- Co-ordination failures - it is often difficult to transfer the knowledge created in universities, etc. to the organisations that can make practical use of it.
Under this rationale, the following objectives were defined:
- to transfer knowledge from academia to business
- to transform the capacity of firms to take advantage of the knowledge transferred on an ongoing basis (embedding greater capability to innovate in future - using placements within the firm versus external consultants).
- (use of placements within the firm) to mediate between business and knowledge producer, to transfer codified and tacit knowledge
- to address the needs of all participating bodies and persons (business, academic, training)
- to provide a supporting infrastructure for the above.
(Regeneris Consulting Ltd., 2010)
Bruno et al. (2011) identify the following rationales for innovation measures in support of science-industry cooperation. These are based on a series of expected effects and underlying hypotheses:
- Overcoming information and behavioural barriers to cooperation between public and private sectors
- Developing stronger channels to facilitate the flow of knowledge and technology from public research organisations to public and private enterprises in a position to deploy that know how and IP in a commercial setting increasing social returns
- Conducting problem-focused research in the expectation that this might expand the total academic effort devoted to user-oriented research and thereby accelerate technological breakthroughs in key areas
- Developing regional capacity (a critical mass of research excellence) in emerging areas with strategic potential as an innovation platform for resident businesses, with strengthening national or regional competitiveness globally.
The latter rationale, developing regional capacity, is more generally a desirable outcome of regionally focused network or cluster policies although it does rely on the aggregated effects of a series of collaborative relationships.
These rationales are more explicitly expressed in the evaluation of the UK Collaborative R&D Programmes (PACEC, 2011), yet they clearly resonate with those noted above:
- To form part of a wider knowledge transfer and innovation support system and brings together key partners from HEls and businesses through interactions focused on projects
- Within this context, to:
- Encourage greater collaboration between businesses and academia
- Support projects likely to result in additional innovation, improve capability and exhibit exploitation potential reflecting the business case
- Address a market failure rationale, based on the premise of a funding gap due to the perceived risk and potential returns on R&D projects for businesses, academics and financiers (potentially with excessive risk aversion) and a lack of information on sources of finance coupled with a lack of understanding of business capabilities to manage R&D and exploit it successfully. (PACEC, 2011)
The reference to addressing market failures (notably risk aversion and informational inequalities) is interesting since policy interventions that promote increased collaboration between actors within the innovation system are typically justified on the grounds of system failures.
The above rationales focus almost exclusively on the promotion of collaboration between the science base (i.e. universities and public laboratories) and the private sector. Few address collaboration between firms.
Martin and Associates et al. (2004), in their review of the literature offer the following benefits for the formation of networks. Since networks depend on the aggregation of a series of collaborative relationships, these benefits apply equally to collaboration between firms:
- Increased scale and scope of activities
- Shared costs and risks
- Improved ability to deal with complexity
- Enhanced learning effects
- Positive welfare effect (increased R&D efficiency and overall R&D expenditure)
- Efficiency (of knowledge transfer)
- Speed (of response to opportunities).
Citing O'Doherty (1998), they note that the benefits of networking can be summarised as follows:
- Material benefits: firms can increase sales and lower production costs by working together.
- Psychological benefits: as firms eliminate their isolation they learn that their problems are shared by others.
- Developmental benefits: By promoting interaction with other firms, networking increases learning and the ability to adapt to the changing economic environment.
It should also be recognised that collaboration can bring risks – and these are factors that must also be considered in the design and formulation of support policies and when considering whether a policy intervention should be developed at all. These risks include:
- "the outcomes do not justify the time and resources invested
- the amount of resources required is under-estimated or under-provided, leaving the collaboration consuming more resources than its benefits warrant
- there is a reduction in flexibility instead of an increase, as the vehicle for collaboration takes on its own set of processes and procedures
- the collaboration drifts away from its original mission and purpose
- there is a loss of autonomy and independence of partner organisations
- because the nature of the collaboration is to work on something new, there is no experience in dealing with problems along the way
- the motivations of the partners to be involved change
- all reputations of the partners are at risk if one partner become embroiled in a scandal or controversy."
(O'Kane, 2008)
It is not the intention of this report to define a comprehensive list of rationales and benefits derived from collaborative schemes. Rather, the examples given above serve to illustrate the significance of the context of the programme within the national innovation system and the perceived policy goals for which it has been designed and implemented.
A final word of caution is that collaboration should not be seen as end in itself: as O'Kane (2008) notes, this can be associated with hollow collaboration (where collaborations are motivated by the desire to access external funds which are then divided up between the partners to obtain an individual benefit) or forced collaboration. The latter can lead to significant inefficiencies: group systems tend to act as levelling agencies, where performance is reduced to a mean level, or even the lowest common denominator. A lack of motivation lead to a lack of responsibility and accountability and the collaboration risks becoming ineffective.
2.1.3 Understanding policy Instruments: targets, governance, and practice
Science-industry cooperation can encompass a diverse set of modalities of action. Bruno et al. (2011) characterise these into two main types:
- Collaborative research centres (CRCs): These are generally located at universities; they tend to focus on specific and strategically important areas of applied research; they often closely engage leading businesses within their governing bodies and in the definition of their strategies. CRCs engage in a diversity of types of interaction and exchange, but tend to exert their most critical impact through shaping research and researcher education (user-orientation) and securing or extending the global intellectual networks. Two main sub-groups may be further defined:
1. Centres of competence (CCs), involving (typically) HEIs (or research institutes) and a consortium of industrial partners
2. Centres of excellence (CoEs), often located within universities; these aim to build a critical mass of competitive research to foster a high level of international visibility. In addition, they tend to focus on research excellence and are hence less industry-driven than CCs although they may aim at objectives related to science-industry cooperation. Most will be awarded longer-term grants of between five and 10 years' duration but are generally time limited.
- Collaborative and knowledge exchange research projects: these entail smaller timescales and scope. Two major types can be further identified:
1. Collaborative research projects: these involve one or more business partners with one or more public research institutions engaged on a specific R&D project of intrinsic commercial value: they are typically co-financed by public grants of 3-5 year's duration, which often covers the costs of the university or public research institute, whilst the private partners tend to pay their own costs
2. Knowledge exchange projects: this covers a miscellany of measures which aim at the support of a specific innovation project, ranging from industrial placements to co-financing of private procurement of technical support services; typically they are very much smaller in scale (e.g. placements, innovation voucher schemes), however, this smallness and flexibility makes them attractive to SMEs.
Since the some of the latter sub-set (knowledge exchange projects) are operationalised through innovation vouchers schemes, which form the subject of another topic report in this series, they are excluded from the analyses in this report.
The 2010 strategic review of the UK Knowledge Transfer Partnerships (KTP) Programme Regeneris Consulting Ltd. (2010) defines a number of types of collaborative arrangements within the UK:
- "The long term embedding of associate within a company to address its specific needs. Also provides academic linkage and training opportunities (for example, KTP).
- Networking initiatives which bring businesses and companies together but fall short of comprising a research project. They provide a forum for initiation of contacts between businesses and/or academics and do not in themselves constitute a research project; typically cover issues of broad interest to a range of businesses rather than those of a single business.
- Initiatives that promote research on a specific issue but which do not include embedding mechanisms. Include projects where independent experts, rather than embedded staff, undertake research on specific business problems, including commercial consultancy or its publicly funded offshoots; shorter term nature of research projects rather than longer term involvement.
- Schemes which bring together a business and research institution but there is no embedded associate to undertake the work (e.g. UK Collaborative R&D grants).
- Initiatives which embed an academic in an organisation, but not primarily to address business needs (e.g. Industry Fellowships, which concentrate on ensuring that the work undertaken by the Fellow (researcher) closely ties in with that of their home institution to ensure their long-term future development). Industry or Enterprise Fellowships do not involve an associate who may remain with the business over the long-term.
- Industrial CASE awards, where the company partner defines the research project, and picks an academic partner; the PhD student undertaking the research fulfils the same role as a KTP associate. There is less focus on business needs (and more on the training needs of the student." (Regeneris Consulting Ltd., 2010)
Bruno et al. (2011) provide a more detail set of characteristics for the operation of competence centres/centres of excellence: “these are long-term but often set for a limited period of time (e.g. maximum six years for the Finnish centres of excellence with funding received in two instalments; five years in the Irish competence centres). The size of such centres varies according to the focus of the centre (e.g. Finnish Centres of Excellence are composed of research units with about 20 to 200 staff). They are either created as a distinct legal entity (Austrian model) or integrated in universities (Swedish Model). They are implemented most of the time under the form of national programmes, most of the time with a regional dimension (e.g. in Austria the Bundesländers (provinces) provide additional funding for the COMET centres of competences and can take part to the decision-making process)".
2.2 Challenges for evaluating policy
2.2.1 Major anticipated impacts
The evaluation of the UK Collaborative R&D (CR&D) programmes (PACEC, 2011) provides a useful example for discussing the potential challenges for evaluating this type of measure. The evaluation report notes the following broad aim of the evaluation: To assess the CR&D, its outputs, outcomes and economic impact, the wider benefits and the lessons that can be learnt for developing similar programmes in the future. Under this, a series of more specific aims are elaborated:
- "To define a set of outcome metrics which provide a useful measure of the full benefits of the programme and are consistent with the Government's expectations of Return on Investment (ROI) evaluation and targets
- To understand and quantify the direct and wider benefits that the programme has delivered
- To ascertain the above at several layers of the hierarchy, including programme, project and sector levels through to partner types (i.e. businesses and academics) and national/international impacts
- To ascertain whether the potential impact has increased with the introduction of the challenge-led agenda to complement the technology inspired areas
- To ascertain whether an increase in support of smaller projects provides greater or lesser benefit To ascertain and quantify the benefits of using CR&D in conjunction with other instruments to deliver strategic and tactical objectivesTo ascertain the influence and impact of aligned partners and funding
- To ascertain the overall strategic value of CR&D in the role of the Technology Strategy Board and in support for innovation in the UK economy
- To identify and recommend where/how the CR&D can most effectively be used."
(PACEC, 2011)
In order to address these aims, the evaluation employed a series of metrics. These serve as a useful example of a typical set of metrics that could be broadly applied to the evaluation of many collaborative support schemes:
- Employment including number of actual and likely full time equivalent (FTE) gross and net additional jobs created and safeguarded
- GVA expressed (gross and net additional), both actual and likely
- Changes in attitudes and behaviour as expressed through interview results on:
1. Contribution to the costs of projects
2. Shared risk of investment
3. Strengthening of collaborative activity: businesses / academics
4. Provision of access to technical and R&D skills
5. Led to leading edge research
6. Improved technical understanding and knowledge
7. Improved attitudes to collaboration
8. Allowed technical feasibility of ideas to be assessed
9. Allowed application of technologies to take place
10. Developed products and processes
11. Produced social impacts
In the Strategic Review of the KTP Programme, the evaluators formulated a key set of metrics relating to the anticipated impacts of the programme. These related to aspects of the programme as follows:
- Operation/management:
1. Management efficiency
- Minimised bureaucracy
- Administrative timeliness & responsiveness
- Administrative & Financial transparency
- Adequacy of support infrastructures
- Performance:
1. Number of active collaborations
2. Efficiency (balance of research to admin costs)
- Impacts:
1. Participant overall satisfaction
2. Address participants' motivations
- Academic engagement with industry
- Generate additionality
- Practical experience
- Training aspects
3. Socio-Economic benefits
- Value of new sales
- GVA
- Social benefits (based on research results)
- (new) Employment
4. Wider economic impacts
- Increased capacity for innovation
- Spill-over effects (to suppliers, etc.)
- Improved business-business relationships
5. Academic benefits
- Feedback into academic teaching
- Identification of new research themes
- New/novel training opportunities
2.2.2 Key evaluation challenges
Clearly, while a number of the outcome and impact metrics described above are relatively easy to monitor and measure either directly (employment, turnover, etc.), or through the use of proxies such as joint publications, co-patenting, and licensing, several, particularly those relating to behavioural changes and longer term impacts (regional economic prosperity/growth, transfer/exchange of knowledge, etc) are much harder to quantify and measure, whilst the more sophisticated elements of knowledge transfer (improved technical understanding, improved image and reputation) may be hard to capture. As noted by Lemola and Lievonen (2008), the measurement of societal impacts is particularly problematic for evaluators.
The evaluation of collaborative schemes share several challenges that are common to the evaluation of other innovation support schemes. These include:
Timing and Periodicity of evaluations
While formal collaborative arrangements are comparatively easy to identify from their outset, several further anticipated outcomes and impacts are not likely to manifest until some time into or even after the completion of the collaborative arrangement. For this reason, the timing of the evaluation will have to strike a balance between being early enough to deliver timely management information and yet allow sufficient time to pass for sufficient results to be generated. Typically, evaluations of collaboration schemes often focus on the operational characteristics of the programmes, thereby providing feedback to managers and scheme administrators in order that the administration of the scheme is streamlined and tailored to the needs of the users. This use of evaluation as a management tool implies that evaluation results should be timely and not be available only towards the end of a project.
Collaborative arrangements develop and mature over time. Moreover, the collaboration project itself often forms the focal point for the continued development of a collaborative relationship that, ideally, can last several years. Such follow-on activities may be difficult to assess within the available time frame of an evaluation – i.e. take too long to develop and will be less tangible and quantifiable. Again, this will pose difficulties for the timing of the evaluation and, depending on the specific objectives of the policy measure, it may be necessary to undertake two or more evaluations at successive stages of the lifetime of the measure: for example, one to address managerial/administration/operational aspects, one to review results and one to address longer term impacts and outcomes.
Scope of impact
Although certain metrics may capture the overall outcomes of a programme, the outputs and experiences of individual projects and relationships may be difficult to discern. Whilst collaborative projects or placement schemes, for example, may have apparently simple outcomes (R&D results (joint publications, patent applications, new prototypes, etc.), trained personnel, the true benefits of collaboration are more complex and involve longer term relationships, or behavioural changes amongst the partners engaged (see below). Hence, quantitative data capture must often be reinforced with more detailed qualitative interview or case study approaches. However, an associated problem also applies - the use of interviews and similar approaches to capture experiential information poses problems for data analysis particularly at an aggregate level, while the use of broad participant surveys may fail to capture important qualitative aspects and unanticipated outcomes. Whilst a broad range of data collection and evaluative approaches may be required, this has clear cost implications.
Counter-factuality and benchmarking
In common with other types of policy intervention, it is difficult to identify the types of outcome and impact that arise from the creation and development of an impact arising from a collaborative relationship in the absence of counterfactual examples or benchmarks established prior to the formation of the collaborations. The identification and selection of control groups which may be used to derive baseline data can be difficult. Likewise, it will be difficult to benchmark the performance of the collaboration in the absence of a detailed ex ante assessment of the conditions prevailing before its initiation.
Finally, as in most evaluations, the issue of attribution is difficult to assess. Although the impact on, for example, student training can be detected since this affects individual in a way that can have a major impact on their career development, the impact of a single collaborative project, particularly on a larger business entity may be much harder to define.
In their review of the UK's academic-industry collaborative schemes, Howells et al. (1998) found that "HEls and industry are also entering into a new and wider set of research and training links, based on partnerships with deeper but less formal relations. This can mean one-to-one collaborations associated with, for example, company-funded laboratories on university campuses, through to more complex sets of inter-firm (and inter-HEI) consortia". A consequence is that formal Government support policies now operate against a much broader background of informal relationships which makes an assessment of attribution and additionality much harder to undertake.
3 Scope
3.1 Methodology
In order to generate a literature base for this report a mix of systematic and non-systematic searches were performed. A first step was to search the existing INNO-Appraisal Repository for examples of collaborative research support programmes. A similar procedure was performed on the database of policy support measures derived from a study conducted for DG Regio into the evaluation of regional innovation support measures.
A second set of searches was performed on the innovation support measures contained in the combined ERAWATCH-TrendChart database. The measures described in this database contain information on whether or not they have been the subject of an evaluation, sometimes accompanied by a source location for any relevant evaluation reports. Since the information in this database is not always fully up to date, additional Google® searches were performed using the title of the measure. In some instances, this procedure revealed more recent evaluation material, including literature associated with the evaluation of the measure and not restricted to the actual evaluation reports themselves. A similar process was also applied to the titles of the measures identified in the INNO-Appraisal and DG Regio studies, in order to ensure more recent evaluations and more general literature were included. A series of Google® searches on specific keywords, such as 'innovation networks' and similar terms was also performed.
The general background literature dealing with innovation policy evaluation and innovation policy interventions was also scanned for examples of relevant evaluations. A sophisticated search using the Scopus database revealed more than 200 articles; after eliminating those that were not relevant to this study and which lacked the necessary quality, around 50 articles remained for analysis. It was observed, in general terms, that a portion of the academic literature is linked to evidence obtained through evaluations, because the scholars were also the evaluators, they advised the policy-makers or evaluators, or they used the data collected in the evaluation process as the basis for academic outputs.
Finally, members of the study team were also familiar with particular examples of evaluations which were brought into the analysis.
3.2 Identified evaluations
A number of potential evaluation reports and associated literature was identified from the above processes. These were screened in order to exclude examples that specifically concerned cluster policies or cooperation policies (using the definition outlined in Section 2.1.1). However, where the evaluation literature dealt with the issue of networking (although the main policy rationale concerned clusters or cooperation), relevant material was extracted.
The set of evaluations identified for analysis is shown in Annex 1.
One word of warning is that, despite the large number of evaluation reports and associated literature that were available on the subject of evaluation, most tend tended focus on the success (or otherwise) of the programmes in achieving their objectives rather than on the reasons underlying the success. Thus, although the question of “what worked?” was addressed, few studies examined the "how?” and “why?" successful outcomes were achieved.
4 Summary of findings
Our review of the literature is structured around three main types of impacts typically created by collaborative measures. Section 4.1 reviews the evidence related to the creation of additional input arising from government intervention (i.e. input additionality), while Section 4.2 discusses the evidence on the outputs that would not have been created without a collaborative policy (i.e. output additionality). In Section 4.3, we outline the evidence related to the change in the behaviour and the process of support (i.e. behavioural additionality). There is a close link between output and behavioural additionality in collaboration schemes. We have included studies investigating only the quantitative increase in collaboration by government support in output additionality (Section 4.2) while studies that look at any persistence or some sort of qualitative change in collaboration as part of behavioural additionality (Section 4.3). In Section 4.4 we review the evidence related to programme design and governance.
During our review it became evident that the scope and quantity of the available literature concerned with collaboration policies were very extensive. In order to focus our search for evidence, we decided to pay particular attention to five important thematic areas of the literature (Section 4.5):
- The evaluation of the Alvey Programme, a ground breaking evaluation of a collaborative R&D programme (Section 4.5.2).
- The prevalence of large-scale R&D collaboration support mechanisms aimed at the development of partnerships and the transfer of knowledge between scientific actors and industry. The selected example was the case of the EU's multinational EUREKA initiative (Section 4.5.3).
- There is a long history of R&D collaboration support. A notable example is provided by the Japanese experience of national level policies for inter-firm R&D collaboration (Section 4.5.4).
- The Community Innovation Survey and other widespread surveys have extensively been used to understand the impact of innovation policy (Section 4.5.5).
- Australia's Cooperative Research Centres Programmes has been praised as one of the most successful examples of collaborative support programmes. The impact of this programme has been investigated in a series of studies (Section 4.5.5).
In our examination of the several evaluations that have been conducted into collaboration schemes, it has become clear that evaluations tend to focus on the extent to which policies have been successful in achieving their objectives rather than how they actually achieve such success. Thus, evidence is frequently presented on what has occurred, the stakeholders' views on the process of participation and what have been the outcomes all questions that focus on the 'what' and 'who' rather than on the 'why' and the 'how'. Consequently, many policy lessons are put forward together with recommendations for the future conduct of the policies by which the programme may be improved. Often these take the form of pre-conditions for success: i.e. address these issues and the chances of success will be improved.
4.1 Evidence of Input Additionality
There are a number of studies, especially in the academic literature, that investigate the relationship between government funding and increased inputs. For instance, Branstetter and Sakakibara (1998), Mothe and Quelin (1999) and Watanabe et al. (2004) found a positive statistical relationship between R&D spending and government funding through collaborative R&D programmes. Czarnitzki et al. (2007) confirms this finding in Finland but found no difference in Germany. In the study by Arranz and Fdez. de Arroyabe (2008), the positive relationship is between support and general spending by the firm. Mercer and Stocker (1998) found that the Australian CRC programme stimulates greater industry spending on R&D and greater industry involvement in guiding R&D in the public sector.
The common methodological issue in these studies relates to causality. They statistically prove a correlation between government support and an input of collaboration. However, in most of the cases these studies do not explain whether the independent variables affect the dependent variable in question or vice versa. Therefore, the question of causality remains open. On one hand, a statistical correlation between government support and increased R&D spending in collaborative firms might show the impact of government intervention. On the other hand, it might be due to the fact that the already collaborating firms have more chance in obtaining government support. Causality is even vaguer in studies that use large surveys without distinguishing specific support programmes (Table 2).
| Reference |
Findings |
| Branstetter and Sakakibara (1998) |
Positive relationship between participating in R&D consortia and increased R&D spending |
| Mothe and Quelin (1999) |
Positive relationship between collaboration and increased R&D spending |
| Watanabe et al. (2004) |
Positive relationship between collaboration and increased R&D spending in other firms (spill-over) |
| Czarnitzki et al. (2007) |
In Finland: Positive relationship between government support and R&D spending and patenting
In Germany: Positive relationship between government support and patenting |
| Arranz and Fdez. de Arroyabe (2008) |
Public funding in general has a positive influence on firms' expenditure |
| Mercer and Stocker (1998) |
CRC programme stimulates greater industry spending on R&D and greater industry involvement in guiding R&D in the public sector. |
4.2 Evidence of Output Additionality
There are a number of studies investigating the effect of government subsidies on collaborative R&D between firms. Kleinknecht and Reijnen (1992), Sáez et al. (2002), Kang and Park (2012), Bozeman and Gaughan, (2007) and Segarra-Blasco and Arauzo-Carod (2008) found a positive relationship between collaboration and government support, Branstetter and Sakakibara (1998) found a positive relationship between participating in government funded R&D consortia and research productivity, Czarnitzki et al. (2007) and Miotti and Sachwald (2003) linked government support and patenting in collaborative R&D projects. In their conclusions to the analysis of 10 evaluation studies of the ATP, Ruegg and Feller (2003) found "considerable evidence that ATP-funded projects generate outputs—publications, patents, patent citations, collaborative linkages, and products that will potentially lead to knowledge and market spillovers". An evaluation of the Australian CRC programme revealed, by means of an econometric modelling approach, that “the Australian economy's overall performance has been considerably enhanced when compared to the performance that would have occurred in the absence of the funding" for the 18 years period prior to 2010 (O'Kane, 2008: 41).
Hughes and his colleagues (Cosh and Hughes, 2007; Hughes, 2008), through their CBR surveys in the UK since 1991, found that the increase in the amount of collaborative activities (especially between university and industry) is due to increased importance given to university industry collaboration in UK innovation policy in 1990s and 2000s. In a series of evaluations of the UK higher education "third stream activities" supported by HEFCE/OSI third stream funding, especially in comparison with these activities in the US, Hughes and colleagues (Abreu et al., 2009; CBR and PACEC, 2009a, b, 2010a, b, c, d). found a positive impact of funding on university and industry links, although they point out some potential dangers that might arise from over collaboration.
Some studies made more precise measurements. According to Mohnen and Hoareau (2003) government funding increases collaboration between firms by 8.1%, for Afcha Chávez (2011) this ratio is 14.2% at the national level and 8.7%.at the national level, for Busom and Fernández-Ribas (2008), it is 28%.
Evaluations were even more precise. PACEC (2011) calculated that the CR&D programme in the UK created 13,350 jobs and GVA of 2.9bn (a benefit cost ratio of 6.71). The KTP programme generated between £4.2 and £4.6 billion of new sales, between £1.6 and £1.8 billion of GVA and 5,530 – 6,090 jobs between 2001/2 and 2007/8 (Regeneris Consulting Ltd., 2010). A study by SQW Ltd. (2002) found that the impact of the Teaching Company Scheme (the forerunner of the Knowledge Transfer Partnerships) was £70m to £73.5m in extra turnover and 470 extra jobs. Ruegg and Feller (2003) summarise the evaluations of the US ATP programme and quote the ratio of 46% to 84% for increased collaboration (Table 3).
The LINK Collaborative Research scheme has, since 1986, been one of the UK government's main mechanisms for the promotion of collaboration in pre-commercial research between businesses and the research base. An independent strategic review in 2003 found that LINK had provided good value for money and had led to substantial economic benefits for participating companies (Smith et al., 2003). Since its inception, it is estimated to have increased the profits of participating companies and raised employment levels by 15,000 to 25,000. It was well regarded by both business and university users.
LINK has also led to substantial direct economic benefits for participating companies. The 2003 review notes that estimates for the "direct" effect of LINK on the UK economy since its inception are:
- between about £700m and £2,400m in terms of increased turnover, and benefits to costs ratios of between 1.1 and 3.8:1;
- between £250m and £500m in terms of increased profit;
- between 15,000 and 25,000 posts in terms of increased employment.
There has been a concern that collaboration between industry and university would be detrimental to universities in terms of research and teaching performance. There are a number of studies that used survey based evaluation approaches to understand the impact of collaboration with industry on universities For instance, Blumenthal (Blumenthal, 2003; Blumenthal, Campbell, et al., 1996; Blumenthal, Causino, et al., 1996; Blumenthal et al., 1986) finds that in life sciences collaboration with industry increases research and teaching output and impact, but after a degree it decreases publication output and increases unpublished work.
There are also a number of publications that employ bibliometric methods to evaluate the publication outputs of university industry collaboration. In their highly influential paper Katz and Hicks (1997) calculated that the impact of publications (e.g. citations) increase dramatically if UK researchers collaborate with industry, especially if the collaboration is international. There is a stream of research using this type analysis which generally tends to corroborate this result (Frenken et al., 2010; Godin and Gingras, 2000; Lebeau et al., 2008; Lee and Bozeman, 2005).
As with input additionality, the issue of causality prevails in the studies that provide evidence on the output additionality of collaborative programmes. Interestingly, this is only openly discussed by Kleinknecht and Reijnen (1992) who warn that their findings reveal "purely quantitative relationships (...) not necessarily identical with causal links”.
| Reference | Findings |
| Table 3: Evidence on Output Additionality | Header 1 | Header 2 |
|----------|----------|
| Data 1 | Data 2 |
4.3 Evidence of Behavioural Additionality
The concept of behavioural additionality was coined in 1995 by Georghiou and colleagues (Buisseret et al., 1995) to complement the traditional measure of input and output additionality. They argued that the fact that a firm spends more on R&D because of government support (i.e. input additionality) or the amount of outputs it creates with the help of government support (i.e. output additionality) are not sufficient to assess the success of a policy or to design a new one. For the first time, they proposed to analyse what happens inside the firm as a result of the government intervention by asking the question "what difference does policy make in the behaviour of the firms it supports?" (Buisseret et al., 1995).
Since 1995, the concept has attracted a considerable amount of scholarly and policy attention. Around half of the innovation policy evaluations in Europe (conducted between 2002 and 2007) investigated the issue of behavioural additionality implicitly or explicitly (Gök and Edler, 2010). The concept has been used with four different interpretations: i) an extension of input additionality covering increased scale, scope and acceleration, etc., of the desired outcomes, ii) the change in the non-persistent behaviour related to R&D and innovation activities, iii) the change in the persistent behaviour related to R&D and innovation activities, and iv) the change in the general conduct of the firm with substantial reference to the building blocks of behaviour. The majority of evaluations and scholarly studies discussing the concept of behavioural additionality use collaboration as one of the key, if not the sole, behaviours on which they focused. This is especially true if one of the first three definition categories mentioned above is used. For instance, in their summary of the nine OECD member country pilot studies of behavioural additionality, Georghiou and Clarysse (2006) define "network additionality" (whether the project would be conducted in a less collaborative way in the absence of the support) as a dimension of behavioural additionality. Eight of these studies reported that between 42% and 70% of the projects involved more collaboration because of the support they received (OECD, 2006).
Some of the academic literature studying the impact of collaboration policies has also explicitly used the concept in the context of industry-industry collaboration. For instance, Davenport et al. (1998: 65) claimed that "the existence of behavioural additionality, while enough in itself to justify involvement in collaborative research, is also likely to strengthen a policy's latent ability to influence the creation of output additionality". Autio et al. (2008) argue that there are effects that influence the target firm's collaborators although they are not direct beneficiaries (i.e. second order additionality). Busom and Fernández-Ribas (2008) provide evidence that public support in Spain increases collaboration both with other firms and public research organisations (i.e. creates behavioural additionality). Clarysse et al. (2009: 1523), in their empirical investigation of the impact of the Flanders IWT programme, argue that inter-organisational learning forms one of the dimensions of behavioural additionality and they show that "the more partners with whom a company puts in a demand for R&D support, the more it learns from the project in collaboration with these partners". Fernandez-Ribas and Shapira (2009) found that innovation policy in Spain created behavioural additionality by increasing the internationalisation of firms' innovation activities and also by changing how innovation activities are organised. Teirlinck and Spithoven (2010) argue that regional funding in Belgium is more influential in creating behavioural additionality by encouraging firms to establish persistent research collaborations with universities and public research organisations. However, they also show that the firms that already collaborate receive more regional and EU funding and therefore the use of "lagged public funding to explain research cooperation has serious shortcomings to measure behavioural additionality through public funding of research (Teirlinck and Spithoven, 2010: online)."
Gök (2010a) argues that although the majority of behavioural additionality studies criticise the evaluation literature of treating the firm as black-box and only looking at what it takes as inputs and what it creates as outputs, they employ the very same logic in the evaluation of behavioural additionality. They attempt to open the black-box of the firm, find smaller black-boxes of behaviour inside and treat these smaller black-boxes according to the input they take and output they create. For instance, most of the behavioural additionality studies measure collaboration behaviour as the change in the resources for collaboration (input of collaboration) or the number of firms with which the firm collaborates (output of collaboration). However, as behaviour as such is not a unit of analysis and this approach compares only two static points, in most of the cases these studies do not succeed in providing evidence on the dynamics of behavioural change. For instance, behavioural additionality studies that look at collaboration rarely investigate the change in i) patterns of collaboration (i.e. if the firm collaborated differently, with some other firm that it would not have done so without support), ii) rate and trend of collaboration (i.e. if the changed collaboration is a persistent one, if the government intervention triggers an ever increasing collaboration or it will gradually decrease sometime after the intervention, and so on) and iii) internal dynamics of collaboration (if the collaboration is extended to other parts of the firm because of the government intervention).
Further evaluation studies look at the behavioural changes brought about through collaboration but do not make explicit reference to the concept of behavioural additionality. For instance, in two different evaluations of the ATP, summarised by Ruegg and Feller (2003), an average of 72% of the participants felt that the collaborative nature of the ATP programme benefited them by stimulating creative thinking, 58% by obtaining R&D expertise, 55% by accelerating entry to the marketplace and 46% by encouraging future collaborations. According to Ruegg and Feller (2003) "ATP funding leverages and accelerates R&D, refocuses R&D on more technically challenging problems and enabling platforms of technologies, and fills a significant funding gap".
There is a stream of academic research that looks at the behavioural effects of university industry collaboration starting from the influential study of (Etzkowitz, 1998). (Rothaermel et al., 2007). This stream sees that the evolution of university started with a teaching-only function and the importance of research within the universities increased and finally the "third revolution" in academia was the transformation to "entrepreneurial university” which has very close ties with industry (Rothaermel et al., 2007).
The issue of knowledge exchange was examined in a number of evaluation reports: in his evaluation of the US State/Industry University Cooperative Research Centers (S/IUCRC) Programme, Roessner (2000) found that the most quoted impact by participants (90%-95%) was that of obtaining access to new ideas or know-how. This impact was quoted by 90% of surveyed participants in the NSF's Engineering Research Centres (ERC) Programme (Roessner et al., 2004).
A recent evaluation of the CR&D programme in the UK reveals a similar set of impacts: "changes in attitudes and behaviour" including strengthened collaborative activity with businesses / academics (84% of the participants), access to technical and R&D skills (67%), leading edge research (59%), improved technical understanding and knowledge (84%), improved innovation, R&D skills and processes (92%) and improved attitudes to collaboration (84%) (PACEC, 2011).
An evaluation of the KTP programme, also in the UK, reveals that as well as obvious economic effects, the programme benefited academic partners by providing insights for academic teaching as the academic partners were able to gain insights to industry practices through the KTP associates embedded in the participating companies. Furthermore, it also gave the academic partners the opportunity to identify new research themes (Regeneris Consulting Ltd., 2010).
Finally, the UK LINK programme was found to represent an important means through which participants could strengthen their research capabilities, diversify their knowledge base and undertake researcher training (DTI, 2003). LINK both strengthened existing relationships and helped develop new ones, around half of the relationships within LINK projects involved new collaborations between organisations. By the time of the review, some 20,000 new relationships had been brokered and many were likely to be exploited well beyond the immediate life of the LINK project. In terms of scientific quality, LINK outputs were found to be broadly comparable, in both quantitative and qualitative terms with other Research Council funded outputs: bibliometric analyses confirmed that a proportion of LINK projects were published in high impact journals, and attracted an above average number of citations from the rest of the scientific community.
Although LINK had no specific objective to act as a researcher mobility scheme, the evaluation found that it made a substantial contribution to the transfer of knowledge and technology between the science base and business through the movement of personnel on average, at least one researcher per project tended to transfer to the business community.
4.4 The Effects of Programme Design and Governance
Several of the evaluations reviewed examined the role that characteristics of the programmes' design and governance processes played in contributing to their success (or otherwise).
The careful construction of collaborative partnerships and consortia appears to have a positive effect on collaboration project outcomes. For example, a study by Dyer and Powell (2001) indicates that the success of particular Joint Venture projects under the US Advanced Technology programme were more likely to be successful where there was:
- Greater trust and information sharing between partners
- Prior experience with working together
- An optimal number of members
- Proximity in location
- Stability in personnel among the joint venture participants
- Sharing of complementary goods and services (vertical joint ventures) as compared to those which include competitors (horizontal joint ventures)
- In some cases, the presence of motivated participants, members who act as product champions, and/or professional project managers.
The lessons drawn from the study were that “ATP contributes to the success of joint ventures by: requiring more upfront commitment from top management; fostering a more goal-directed and organised project through ATP's demanding application process; and working with joint ventures through difficult periods in their life cycle while helping them overcome barriers to collaboration.
Similar factors were identified in an evaluation of the US Engineering Research Centres Programme (Roessner et al., 2004). The five factors rated as "very important" or "extremely important" in contributing to the benefits a company derived from ERC participation were:
- A strong ERC "champion" in the company unit
- Management support of the ERC within the company
- Alignment between the ERC's specific technical focus and that of the company
- Responsiveness of ERC faculty/researchers to the company's needs; and
- The ERC's efforts to communicate and stay in contact with sponsors.
The issue of management also formed one of the recommendations of the 2010 strategic review of the UK Knowledge Transfer partnerships (KTP) scheme, which called for a stronger management structure based on a clearer division of responsibilities backed up with reduced levels of bureaucracy which would deliver a more efficient programme (Regeneris Consulting Ltd., 2010). In addition, according to Wilson (2012), one of the reasons underlying the success of the KTP scheme is that it is an intensive intervention with a highly structured management, which ensures that it is suitable for businesses with low absorptive capacity: most of the KTP business partners are SMEs. Nevertheless, the 2003 Strategic Review found that it also suffered from an overly bureaucratic application process, and improvements were put in place to reduce the time from application to project start from 52 to 22 weeks. Likewise, lengthy negotiations over intellectual property were found to be an impediment but were mitigated by the introduction of a model contract for all partners.
An interesting finding by Ruegg and Feller (2003) was that collaborations within ATP were frequently fluid and that the composition of partners could fluctuate within the project life cycle. The effects of such changes could be neutral, with no change to the ATP criteria, or negative with the loss of key participants or a departure from the more challenging research goals. In the latter case, intervention by the ATP management was required, underlining the need to monitor projects on an ongoing basis to enable response to and management of change.
Through a process of careful design, planning and strong (ex ante) management, it was found that ATP had been notably useful in accelerating the development of high risk technologies, increasing project stability, getting projects through particularly difficult periods in their life cycles, overcoming barriers to collaboration, and increasing up-front planning. It could be argued that this was not a direct outcome of ATP funding per se, but the result of careful ex ante participant selection processes: it was found that ATP was successful in selecting projects with attributes conducive to generating large knowledge spillover effects. For example, the ATP selection process favoured projects whose (partner) firms had more extensive ties to other businesses, and, hence, were better positioned to realise commercial success and related market spillovers (Ruegg and Feller, 2003). This point was echoed in the 2010 strategic review of the UK KTP scheme where the reviewers noted that, in the selection of projects there should be "more targeting on priority partnerships which generate the greatest impacts and returns"
(Regeneris Consulting Ltd., 2010). Careful planning was also noted as a contributory factor to the success of the Greek Programme for the Development of Industrial Research – PAVE: the effectiveness of the programme was found to be linked to the success of applications, which in turn were positively affected by: "an emphasis upon documenting the technological, commercial and productive importance of the specific research to the company's development, the innovative character of projects and the social and environmental benefits anticipated, coupled with an obligation to submit a commercialisation plan for the research results within a specific time period after project completion" (European Commission, 2003). An effective administration, management organisation, and clear monitoring and evaluation were again identified as success factors.
Thus, there are clear lessons that selection processes should try to ensure that potential consortium or collaboration partners meet a set of criteria that are likely to increase the probability of a successful collaboration outcome – 'forced marriages' do not work – and that there is a need for continual and strong management of collaborative partnerships.
A caveat is that the implementation of strong management and good administrative practices (including ongoing monitoring processes) should be provided without recourse to overly burdensome and bureaucratic approaches and requirements. Criticisms relating to the requirements and time involved in the application and approval process, delays in recruiting participants and in making payments, etc. are commonplace across the recommendations contained in several of the evaluations studies reviewed. Unfortunately, this seems to be a typical disadvantage of collaborative programmes (as noted in the US ATP by Ruegg and Feller (2003) for instance, while the 2008 review of the Australian Cooperative Research Centres highlighted the overly robust system of performance assessment system of monitoring and reviewing and called for it to be moderated (O'Kane, 2008).
4.5 Evidence on Selected Thematic Areas
4.5.1 Evaluation of the Alvey programme
The Alvey Programme, a five-year collaborative R&D Programme in Information Technology, began in 1983. Its aim was to support the UK's academic and industrial science and technology base, transfer academic know-how to industry, and enhance the competitive potential of the sector. It focused on pre-competitive R&D with some elements nearer to market and was the single largest UK IT policy initiative of the 1980s, with a planned public investment of £200m and a further £150m industry contribution. Over 300 R&D projects were initiated – almost 200 of which involved both industrial and academic research teams (Guy et al., 1991).
While this is not explicitly a thematic area of evaluation, the novelty of the evaluation approach, which employed real-time evaluation methodologies such as co-nomination analysis, the size of the programme and the fact that it was one of the earliest large-scale programmes to be independently evaluated by a team of external experts makes the evaluation of the Alvey programme a significant milestone, not just in the evaluation of collaborative programmes, but also in the evaluation of innovation support mechanisms, hence its inclusion here.
The evaluation did not explicitly seek to determine the impacts of the programme, rather it addressed the extent to which broader objectives had been achieved and sought to draw lessons for a range of stakeholders engaged in the programme (industrialists, academics, programme administrators, policy makers). The evaluation pointed to a number of technological, structural and strategic achievements, such as whether project goals were met, impacts on the size of the population of IT researchers in key areas, etc. Of relevance here, it was found that the programme had been "particularly successful in nurturing links between academia and industry. It primarily allowed industrial teams to benefit from the involvement of academics but was less successful in as a means of allowing academics to access industrial expertise or of sharing know-how between industrial participants. One of its major successes was in fortifying the UK IT research community: it built up capability in key strategic areas. Collaboration was generally beneficial although in some cases, the high overheads of collaboration outweighed the associated benefits, especially for smaller projects and smaller firms".
Overall, Alvey was felt to have represented value for money: the cost-benefit ratio was favourable for the great majority of academic research teams and for about three-quarters of the industrial teams. The following are some of the more generic lessons that emerged relating to the effectiveness, efficiency and appropriateness of collaborative R&D programmes:
Industrialists:
- Need for tight coherent strategies to enhance exploitation prospects.
- Allow flexibility to control for partners' outputs
Academics:
- Further efforts needed to bridge cultural gap between academic and industrial research practice - build on mutual trust and understanding.
- It was possible to maintain on-going basic research despite focus on applied end of the research spectrum.
Programme Administrators:
- More flexible funding arrangements would allow inclusion of greater variety of projects and institutions.
- Effective single budget source for all participants would prevent delays.
- Need for standard basic collaboration agreements and scope for academics to hold IPR would improve exploitation prospects.
- Project clusters provide coherence and pull-through for research.
Policy-makers:
- Support for pre-competitive R&D is a necessary but insufficient means of enhancing the competitive performance of an entire industry sector.
- R&D support should at the very least be accompanied by complementary action to address skills shortages.
- Collaboration, per se, is not a panacea.
4.5.2 Eureka's Impact: The Iceberg Model
The Eureka initiative was established in 1995, initially by 18 countries and the European Commission "to raise, through closer cooperation among enterprises and research institutes in the field of advanced technologies, the productivity and competitiveness of Europe's industries and national economies on the world market and hence strengthen the basis for lasting prosperity and employment" (EUREKA, 1985).
Between 1995 and 2012, Eureka hosted 4,198 projects worth approximately €30b with participants from 40 countries (EUREKA, 2012). It differs from nationally funded programmes or other multinational programmes in the sense that: i) it is decentralised (application is made to a National Coordination Point (NCP) who is also responsible for project selection), ii) it is a distributed coordination mechanism (NCPs together with the help of the EUREKA Secretariat coordinate projects and other activities), iii) it is organised in a bottom-up way (firms initiate the process), iv) its management is very flexible (projects can be of any size, reporting obligations are minimum and the application cycle is fast); and v) it is nationally funded (as such there is no common pot or fixed budget allocation). Eureka has four instruments:
- Individual projects: business partners from at least two member countries conduct a market oriented R&D project;
- EUROSTARS: similar to individual projects but aimed at SMEs (established in 2008);
- Clusters: An average of 20 business partners work on a generic technology (established in 1999); and
- Umbrellas: Unfunded networking activities to generate individual projects or EUROSTARS.
An evaluation in 1991 established that i) "Eureka has made a genuine difference in the overall European R&D effort by motivating more firms to engage in collaborative R&D and creating new links between firms", ii) projects led by SMEs experienced more contractual difficulties as SMEs were not experienced in negotiating Memoranda of Understanding and had little understanding and anticipation of IPRs, iii) "Eureka's loose rules on information exchange between governments mean that many project leaders [do not know the] level of resources each of their partners brings to their project", iv) the additionality of projects led by SMEs is considerably higher than those led by larger firms, v) for SMEs, non-financial resources such as information on emerging standards and prospective partners were more valuable than financial resources (this issue was also confirmed by the evaluation of the UK's participation in Eureka (DTI, 1995)), and vi) while some firms benefit from its flexibility and bottom-up structure, other firms wanted clearer rules (Peterson, 1993: 252-262).
In their evaluation, Mothe and Quelin (1999; 2000: 595, 600) find that i) those who have more resources and R&D collaboration experience gain more benefit from Eureka, ii) if firms engage at an organisational or managerial level rather than purely financial, they benefit more, iii) "the firm's role in the consortium influences both the quantity and the type of resources created: consortium leaders or technological integration managers seem to create significantly more resources than do the other partners" and iv) technological and commercial improvements by the programme manifested themselves not only as "significantly improved hardware and equipment" but also as "new working methods and routines" (i.e. as behavioural additionality (Gök, 2010b)).
Georghiou (2007) reports that, prior to 1996, the Eureka evaluation system was based on a very detailed technical project final report. Due to decreasing response rates and other considerations, a Continuous and Systemic Evaluation (CSE) system was adopted in which socio-economic impact is the main concern: a final project report investigating impact was followed by further short and focused questionnaires, at one, three and five years after. A panel interpreted the results and drafted of the Annual Impact Reports. According to Georghiou (1998: 21) while evaluations conducted before 1996 "have been broadly positive about the effects of the programmes, they have been unable in most cases to deliver the type of information about return-on-investment which some policy makers desire. This is not caused by the inadequacy of the evaluators, but rather because an ROI approach usually presumes a linear/sequential model of innovation whereby the benefits of a single research project lead to or are captured by specific innovations. This is rarely the case, particularly for collaborative research." In 2006, this approach was also replaced due to the difficulties associated with self-quantifying and attribution of impact created by the firms and the high level of subjectivity and variation of results.
The new system of evaluation established in 2006 included an expert panel to conduct detailed case studies of high impact projects: previous evaluations had stressed the skewed nature of Eureka's impact (8% of participations accounted for 70% of economic effects according to Georghiou (1999). Simplified questionnaires were also introduced. The new system was based on Georghiou's “Iceberg Model” which stresses that impacts such as sales of innovative products and reduced process costs are the tip of the impact iceberg and one must also consider impacts such as licence income, use of technology in other parts of the business, new contacts/networks and prestige, organisation and method learning, competence and training, and spillovers to non-participants. This demonstrated issues that were not captured by previous evaluation approaches such as i) Eureka's financial contribution is more important for the firms than its label, ii) for small firms, support from the NCP is more assistive than financial resources and iii) for large firms the cluster projects acted as a platform for standardisation (Georghiou, 2007).
Figure 1: The Iceberg Model
An internal evaluation in 2004 (EUREKA, 2004), on the basis of previous internal and external evaluations (including Sevon and Kreiner (1998)), outlined six key success factors for a Eureka project:
1There should be obvious market opportunities for all the partners involved.
2The project should have clear objectives shared by all partners from the outset and all the activities undertaken should be related to them.
3Finding the Right Partners: all partners should complement each other vertically or horizontally, and they should all add value to the partnership.
4Project Management: There should be adequate structure, communication channels and leadership in place to accomplish the objectives of the project
5Resource Availability: There should be adequate financial, technical and human resources available in each partner.
6Partnership Agreement: Each partner should agree on a Memorandum of Understanding and Confidentiality Agreement from the very beginning.
Representing a step backwards from the Iceberg Model developed by Georghiou (2007), more recent studies focus on the statistical relationship between Eureka participation and general firm performance. Bayona-Sáez and García-Marco (2010) based on Benfratello and Sembenelli (2002) found that it has a positive impact on firms' competitiveness. This finding is also consistent with Eureka's own claims for the creation of competitiveness based on Annual Impact Reports (EUREKA, 2005a, b, 2006), extra growth in employment and sales (E.G.P Applied Economics, 2011), input additionality and extra turnover (EUREKA, 2011) in general and also specifically in Denmark (Danish Agency for Science Technology and Innovation, 2011).
4.5.3 Japan's Long History of Collaborative R&D Policies: The Principal Instrument of Industrial Policy
Collaborative R&D has been seen as the main and most important tool of research and industrial policy in post-war Japan. The aim of Japan's industrial and research policy in the 1960s was "to catch up with western technology, set standards, and promote R&D among small- and middle-sized industries. The 1960s witnessed the growth of R&D consortia and the amount of government support, while policies in the 1970s were either mission oriented (i.e. tackling the increasing oil prices by promoting large scale and collaborative alternative energy and energy efficiency research) or aimed at catching up with the US in electronics and ICT. In the 1980s, the creation of basic, inter-industry, state-of-the-art technologies by R&D consortia formed the objective of several ministries (Sakakibara, 2001).
One of the most famous of Japan's collaborative R&D programmes was the very large-scale integrated circuit (VLSI) project, which aimed at catching up with the US electronics sector in mid-the 1970s. According to Sakakibara and Cho (2002), the reasons behind the success of the VLSI project were i) policy makers had a clear sectoral focus and ii) there was a shared willingness among the participants with regard to collaboration and competition. These factors led to increased R&D cooperation, knowledge-sharing and substantive scale economies.
Although these R&D collaboration policies were very succesful in Japan, Sakakibara (2001: 1014) argues that that the public funding itself “was not sufficient to induce firm participation in R&D consortia as firms tended to participate only if there were complementary economic justifications, especially the growth motive, cash flow, the building of strategic assets, and transaction costs considerations." Similarly, Sakakibara (1997: 447) noted that the overall effects were modest and reported that the "participants do not perceive R&D consortia to be critical to the establishment of their competitive position". According to Branstetter and Sakakibara (1998) there was a small but positive statistical link between governemnt support and the amount and producitvity of Japanese firms' innovative activity. In line with the theoretical literature, Branstetter and Sakakibara (2002: 157) found that R&D spillover potential was positively linked with the outcomes of R&D consortia - this link was stronger if the research was basic rather than applied - "the design of a consortium seems to be more important than the level of resources expended on it in terms of explaining research outcomes."
Sakakibara and Cho (2002) argue that Japan's highly successful R&D collaboration policies were imitated by Korea 15-20 years later, but they failed for a number of reasons. First, the chaebol industrial structure in Korea reduced competition to a personal level and thus made cooperation between large conglomerates almost impossible. This was the case in the zaibatsus of pre-war Japan, however in post-war Japan, the less centralised and less personalised structure of the keiretsus allowed efficient cooperation. Secondly, a 40 year history of cooperation in Japan, in contrast to the much shorter experience in Korea, made cooperation easier. Employing game theory, Sakakibara (1997) shows that repeated instances of cooperation gradually eliminates the tendency for cheating. Thirdly, the experience with R&D was much longer in Japan than in Korea. Fourth, Japan's keiretsus were mutually complementary rather than overlapping, particularly in terms of their know-how and therefore benefited from the process of cooperation and sharing of heterogeneous knowledge. In contrast, Korea's chaebols overlapped each other and therefore had little incentive for R&D collaboration. Finally, the deep mistrust between the chaebols and the Korean government made R&D collaboration programmes less successful than in Japan where the relationship between industry and government is based on mutual trust.
The Community Innovation Surveys (CIS) are a series of surveys conducted in EU member states, EFTA countries and EU candidate countries by the national statistical bodies in cooperation with EUROSTAT. The data collection methodology is based on the Oslo Manual (OECD and EUROSTAT, 2005) and therefore sensu lato compatible with the national innovation surveys conducted in the US, Canada, Australia and other OECD member states. The first CIS took part in 1992, CIS2 in 1996, CIS3 in 2001, CIS4 in 2005, CIS5 in 2007 and CIS6 in 2009. CIS6 was sent to 28,000 UK enterprises with 10 or more employees and the UK response rate was around 50% (BIS, 2012). Among other questions about the characteristics of business and their innovation activities, in two separate questions CIS asks i) if they co-operate on any innovation activities with various types of organisation, and ii) if they received any public financial support for innovation activities from various levels of government.
The first question has been utilised in many studies investigating the issue of innovation collaboration (see Tether (2002), Miotti and Sachwald (2003), Belderbos et al. (2004) and Frenz and Ietto-Gillies (2009) for the most seminal examples). Other studies have investigated the correlation between the first and second questions (i.e. if government support influences innovation collaboration). For instance, Czarnitzki et al. (2007) looked at the correlation between government support and innovation spending and innovative activity in Germany and Finland by using CIS2 data and found that the relationship is positive in Finland and inconclusive in Germany. Miotti and Sachwald (2003) found that public funding impacts patenting positively in collaborative R&D projects in France by using CIS2 data. Mohnen and Hoareau (2003: 142) found that "government supported firms have an 8.1 percentage points higher probability to collaborate with universities/government labs than firms without government support" in France, Germany, Ireland and Spain. Arranz and Fdez. de Arroyabe (2008) found that government support increases the R&D expenditure of collaborative firms in Spain. According to a CIS3-based study conducted by Segarra-Blasco and Arauzo-Carod (2008), while EU, national and regional funding increases the propensity to collaborate, this effect is weaker for regional funding in Spain. In their analysis of Spanish CIS3 data, Fernandez-Ribas and Shapira (2009) reach the conclusion that firms receiving public support have a 4% higher probability to collaborate with non-EU foreign partners and an 8% higher probability to do so with an EU partner.
The validity of the CIS has been under academic scrutiny from the outset and this has allowed a continuous improvement in the survey methodology over time (OECD, 2007; Smith, 2004). However, there are still important reasons why CIS data should be used carefully especially in investigating the impact of government support. CIS does not ask for information on the particular programmes from which firms benefited but asks only if, and at level, they were supported. Furthermore, CIS data is anonymous and therefore it is not possible to conduct a follow-up survey or a qualitative research programme on the basis of the analysis of CIS data. For these reasons it is almost impossible to make any contextualisation with regard to the nature of the government intervention.
4.5.5 Australia's Cooperative Research Centres Programme
The Cooperative Research Centres Programme was established by the Australian government in 1991, with the primary aim of encouraging collaboration in research and development between the private sector and public sector research bodies. Ancillary aims included the development of research concentration for world-class teams of researchers and the preparation of PhD graduates for non-academic careers. In the light of a succession of reviews and evaluations, the CRC Programme's 'mission' has shifted to the support of "end user driven research collaborations to address major challenges facing Australia. CRCs pursue solutions to these challenges that are innovative, of high impact and capable of being effectively deployed by the end users". Since its establishment there have been 14 CRC selection rounds, the most recent being completed in 2011. Over 190 CRCs have been funded by the Australian government and, as of 2011, there were 44 CRCs operating operating across four sectors: agriculture, forestry and fishing (11); mining (4); manufacturing (5); and services (24). During the lifetime of the Programme, over Aus$3.4 billion has been provided in government funding, while CRC participants have committed a further Aus$10.9 billion in cash and in-kind contributions.
Each CRC is "an incorporated or unincorporated organisation, formed through collaborative partnerships between publicly funded researchers and end users. CRCs must comprise at least one Australian end-user (either from the private, public or community sector) and one Australian higher education institution (or research institute affiliated with a university)”. It differs significantly from other public programmes having similar aims by the level of funding (between Aus$12m to Aus$30m), the timeframe of commitment (typically seven years) and in that it requires the CRC participants to establish a formal joint venture partnership, in contrast to the gift-based (or unrequited) relationships that underlie many other research grant programmes (Howard Partners, 2003).
According to a 2004 OECD review, the CRC Programme has "proved an inspiring model for subsequent initiatives in several OECD countries" and, as evidenced by the results of periodic and thorough evaluations, forms an "example of good practice in the field of public-private partnerships for R&D" (OECD, 2004). Overall, it has "promoted a gradual "change in the research culture" in both the private and public sectors, leading to an increased collaboration among researchers and the users of research, in many technological fields, and more generally an improved translation of research outputs into economic, social and environmental benefits to Australia". This success has been attributed to a number of factors relating to the design and management of the programme.
Notable reviews and evaluations include those by: Myers (1995), Mortimer (1997), Mercer and Stocker (1998), Howard Partners (2003), Allen Consulting (2005), Insight Economics (2006), Productivity Commission (2007) and O'Kane (2008).
The 2004 OECD review summarised the important characteristics of the CRC Programme identified by the reviews that had been undertaken up to that time:
- A sound application of the public-private partnership concept to address clearly identified systemic failures in the Australian innovation system.
- A broad and diversified portfolio of CRCs, which in particular targeted technological areas where missing/weak links between research performers and research users had been identified, plus actors ready to use the new institutional framework to bridge these gaps.
- A long-term (open-ended) commitment by the Government.
- CRC Programme funding represented a substantial increase of R&D funding in Australia, which facilitated the mobilisation of all relevant public actors.
- Inclusion of an educational objective, aiming at producing post-graduate students with greater knowledge and a facility (e.g. in the field of project management) to work with research users.
- Equal emphasis on all four major programme objectives: research, collaboration, education, application of outcomes.
- Consistent, transparent and open application and selection process, with clearly stated objectives and criteria. Requirement of applicants to get agreement from all participants prior to the application for funding.
- Requirement for all parties to make a formal, contractual commitment of resources to the CRC for the full contract period (normally seven years).
- A clear management structure providing the CEO with full control and responsibility over all CRC resources, not just CRC Programme funds.
- Requirement of a board representing the stakeholders and chaired by an independent industry/research user representative.
- Monitoring and review in collaboration with centre management through the life of each CRC, identifying issues early and allowing early remediation of identified issues.
- A balance between the acceptance and encouragement of diversity to suit different circumstances with the promotion of uniformity and cohesion between all CRCs, particularly in terms of public relations and dealing with the Government.
- Attention to management issues (for such large ventures involving a substantial number of partner organisations with often competing goals). The CRC Association plays a particularly valuable role in this context through sharing of information, workshops, etc. (OECD, 2004)
Howard Partners (2003) also found considerable scope for improvement in the administration of the Programme to improve efficiency and flexibility but noted that "CRCs have performed a vitally important role in transforming publicly funded discoveries and inventions into products and businesses that are "investment ready"" and had "made a major contribution to the development of Australia's public sector research capacity in areas of national need and global opportunity". Due to the diversity of activities supported, the review was less certain whether it had produced research of an excellent standard that would not have been undertaken otherwise: industry took the view that the programme was too 'researcher' oriented (suggesting low additionality) but CRCs were also viewed as a vehicle for excellent research of relevance to potential end-users (thus additionality in terms of research excellence of relevance was high). Similarly, the focus of the Programme had been on research at the expense of commercialising and utilising intellectual property.
The Howard review (Howard Partners, 2003) was positive on the CRC Programme's ability to enhance collaboration among public and private researchers, and between public researchers and commercial or community interest: it found very positive indications in relation to collaboration, while the CRC arrangements were highly regarded for the networking activities and opportunities provided to researchers. It also noted that one of the most positive aspects of the CRC Programme was its contribution to the training of PhD students and their close involvement and interaction with industry.
Two studies (Allen Consulting, 2005; Insight Economics, 2006) focused on the economic impacts of the Programme. Using a modelling approach, the key finding of the first study was that over the (modelled) 1992 to 2010 period the overall performance of the Australian economy was considerably enhanced when compared to that which would have occurred in the absence of the government investment provided to the Programme between 1992 and 2005. Specifically: GDP was cumulatively (in 2005 dollars) Aus$1,142m higher than if the money spent on the CRC Programme had instead gone to general government expenditure (which would have itself contributed to GDP). In 2005, (modelled) GDP was Aus$143 million higher than it would have been in the absence of the CRC Programme (compared to expenditure of Aus$113 million in that year).
The study by Insight Economics (2006), which did not use a modelling approach, found that for each dollar invested in the CRC Programme (rather than left with taxpayers):
- Australian GDP was cumulatively Aus$1.16 higher than it would otherwise have been.
- Total Australian Consumption was Aus$1.24 higher than it would otherwise have been (Private Consumption was Aus$0.10 higher and Public Consumption Aus$1.14 higher).
- Total Investment was Aus$0.19 higher than it would otherwise have been.
In 2007, the Productivity Commission (2007) reported on the excessive and burdensome costs of compliance and administration, concerns about the substantial financial and in-kind commitments required of CRC participants and about the inflexibility of the seven-year Programme structure.
In addition to restating the Programme's successes, O'Kane (2008) also highlighted some of its (previously identified) shortcomings, such as: the complexity and cost of its governance arrangements, including the high costs of bidding for CRCs, their associated transaction costs, lack of flexibility in matching governance and management to the needs of the partners, and the lack of an adequate return on investment for partners, especially when the CRC is incorporated. It recommended more flexibility and improved efficiency in the selection and review process, and for the performance assessment system of reporting and reviewing to be made less onerous. However, overall it concluded that the Programme was still appropriate, primarily by addressing a gap left by the market directly, and also indirectly, through the influence it has had on the design of other programmes. It also concluded, based on recent evidence from other reviews, that the Programme provided a positive return on government investment. On its enhancement of national growth, O'Kane (2008) was more cautious noting that high levels of commercialisation had not occurred although there was evidence of benefits arising from industry application of CRC research.
In conclusion, O'Kane (2008) stated that, once their recommendations had been taken up, the CRC Programme would continue to maintain its relevance and importance the Australian innovation system and would represent a sound return on the Commonwealth's investment, thereby contributing to a "sustainable, community-oriented, productive, creative and prosperous Australia".
5 Lessons and Conclusions
The literature that has been reviewed in this report has focused primarily on the degree of success that the measures and mechanisms concerned have had towards achieving their objectives. Thus, there is a body of evidence on success factors - what has worked and what has not. However, as already indicated above, less attention has been paid to the underlying issues of why and how various approaches have worked, and in what contexts the likelihood of certain impacts may be improved. In addition, the term 'impact' is rarely used: instead, 'success' is used extensively to imply the attainment of objectives but also to encompass outcomes such as participant satisfaction, the achievement of motivations and effective delivery of the programme, for example. A broad overview of typically encountered success factors (i.e. pre-conditions for successful programme outputs outcomes) is provided by Dyer and Powell (2001) as summarised in Figure 2
Figure 2: Key Factors Influencing Collaborative Success
Starting on the left, the first set of circles contains five factors that participants identified as influencing their success in collaborating. Moving to the far right, the highlighted circle lists the main measures used by the participants in measuring their success. The center of the figure shows the key mechanisms through which the five factors on the left influence the success measures on the right. These factors are trust, information sharing, and coordination costs. The plus and minus signs along the directional lines show the nature of impact in the direction indicated. For example, geographic proximity of the members is shown to contribute to success in two ways: a positive effect on information sharing contributes positively to success; a negative (reducing) effect on coordination costs also contributes positively to success because the costs are negatively associated with success.
Source: Dyer and Powell, Determinants of Success in ATP-Sponsored R&D Joint Ventures: A Preliminary Analysis Based on 18 Automobile Manufacturing Projects, 2001, p. 11.
Many of the recommendations encountered in the evaluation reports examined in this study tend to focus on aspects of programme management, administration and planning. A number of collaborative mechanisms are covered with a high level of variation in the objectives of the programme, the form and conditions of support provided, the intended collaboration participants and actors, and the nature of the collaboration activities supported: in every case, the process of collaboration itself has been treated as a "black box". It has thus been difficult to define a set of variables relating to the collaboration process itself and to indicate which attributes of the various policy instruments concerned work and which do not. For these reasons, it is only possible to generate a set of general lessons for the design and implementation of collaborative support instruments, i.e. the typical pre-conditions for success.
5.1 Programme design
1Some of the most successful measures are seen to be associated with a long-term, stable commitment of government funding and support. Within this, phased funding cycles appropriate to the nature of the collaborative activities undertaken can be established (i.e. support for graduate placements, project duration, operation of a centre, etc.) [e.g. the Australian Collaborative Research Centres Programme].
2It is important to clearly define the purpose of the programme in advance: a stated overall goal in several of the measures reviewed was to stimulate R&D expenditure by industry and outcome metrics were defined according to this goal (for example, as input metrics (R&D expenditure) or as output metrics such as joint publications, patents or new products). Thus, collaboration is often viewed as a means to an end. However, the real benefits arising from collaboration are often intangible and do not lend themselves to simple metrics. For these reasons, the programme (and its associated assessment processes) should be designed to maximise the benefits and outcomes of the collaboration process itself rather than simply on the (anticipated) outcomes that are easiest to measure. [E.g. Wilson (2012); O'Kane (2008)]
3It is also important to recognise that there are many reasons for which participants may wish to collaborate increasing R&D spend is generally not foremost among them. Collaborators may have several objectives (not all of which may align with those of policy makers, although they may have equal validity). These may include the desire to undertake activities connected with education, training, skills development, knowledge exchange, enhanced networking (horizontal or vertical), or longer term outcomes. Similarly, the opportunity to participate in collaboration activities may be seen as an opportunity to experiment with new forms of collaborative arrangement. Thus, programmes must be designed to allow participants to satisfy their motives and maximise their expectations rather than imposing strict operational criteria on them. [E.g. Wilson (2012)].
4For several reasons, such as changes in framework conditions, changes in the policy landscape or shifts in policy priorities, it may become necessary to adjust the underlying rationale and objectives of a programme. Programme managers must be sensitive to these changes and be prepared to adjust the configuration of the programme and, at the same time, to reconfigure the ways in which performance is judged. [E.g. the experience of the US ATP, EUREKA, UK LINK and Australian Collaborative Research Centres programmes].
5Collaborative activities do not proceed in isolation. Their tangible outputs (new ventures, follow-up projects, scientific discoveries, new prototypes or products, trained personnel, graduate students, etc.) may lead to further demands for policy support. Thus, it is important to align collaborative support programmes within a broader supporting and complementary set of policies which can capitalise on their outcomes (both tangible and intangible). (for example, Lambert (2003)].
6Where it was relevant (i.e. in science-industry collaborations), the impact of the programme was found to be enhanced through the inclusion of an educational objective, such as the production of post-graduate students who were able to gain experience of project management in an industrial context. [E.g. KTP programme].
5.2 Selection of participants
7In several cases, programme success was found to be closely aligned with the characteristics of the participants. For example, companies (and some science partners) with certain characteristics (above average performance, already well-networked, experienced with collaboration, strongly motivated, etc.) tended to be associated with successful collaborations. This raises the issue of value-added: if policy support is given to those recipients who are already beneficiaries of collaboration, what additional benefits are gained in contrast to encouraging new partnerships to form? The ideal approach appears to be one that is able to combine a mix of experience and novelty – i.e. matching partners with a track record of collaboration with new partners or defining new areas and topics for collaboration. This has several implications:
- Carrying out careful due diligence on programme participants and their proposed partnerships
- Paying careful attention to ensure an alignment of participants' interests
- Understanding the motivations of all parties to the collaboration – there should be a clear demand for the outcomes of the collaboration
- Ensuring commitment to the full responsibilities of participation and to the successful completion of the joint activities involved. In most instances, the requirement of a financial contribution from industry partners was seen as a means of (literally) increasing buy-in, whilst the use of student placements was a common means of ensuring the engagement of academic partners. Some programmes required that all parties made a formal, contractual commitment of resources to the project for the full contract period. The use of sanctions for non-compliance did not appear to be reported.
- Identifying the people who will be responsible for making the collaboration work - their motivations, rewards and incentives and ensuring that clear leaders/champions have such a role
- Ensuring that, where relevant, applications contain explicit plans, with clear milestones and anticipated outcomes (that are not restricted to outputs alone but which identify process outcomes as well)
- The need for a consistent, transparent and open application and selection process, with clearly stated objectives and criteria. There should be a requirement to get agreement from all participants (not just the lead proposer) prior to the application for funding. [Various programmes, notably the Australian Cooperative Research Centres Programme].
5.3 Programme governance
8The management of collaboration initiatives is a major determinant of success, particularly in collaborations which involve novel partnerships, new research topics or where the anticipated research outcome cannot be guaranteed or where the potential for unexpected outcomes is high. Therefore, it is important to instigate clear planning processes, which include the identification and framing of goals, development of milestones and stages, managing the different phases of a collaboration. This process can start as part of the selection criteria (see 7f, above) but must be continued through the lifetime of the collaborative project, and must be performed internally (within the collaboration, but according to a framework established by the programme management). Such procedures also allow early intervention by management, where necessary and minimise the chance of failure. [For example, Centres for Research-based Innovation (SFI) and the Australian Cooperative Research Centres Programme].
9Following on from the last point of 8, above, a degree of flexibility must be provided in recognition of the fact that each collaboration can be different and that partners may have differing motivations and expectations.
10In long-term collaborative programmes it is important to define clear exit points and processes to allow for changes in partners and participants to be effectively managed whilst avoiding high turnover rates and the sudden loss of key partners. Typical approaches towards this include phased funding cycles and financial contributions from the partners. [For example, Australian Cooperative Research Centres Programme].
11Effective collaboration depends on mutual trust – that workloads will be shared equitably, that all benefits will be shared and that shared information will remain within the confines of the partnership. While careful selection processes can enhance the environment for trust, it is essential that governance conditions also promulgate trust. This may include establishing rigorous and transparent conditions for the ownership of the outputs of the collaboration such as intellectual property. However, a strong IPR framework in itself is not a substitute for trust engendered through mutual respect for the partners. [Several evaluations point to the issue of trust, in both firm-to-firm and science-industry collaboration programmes].
12Strong governance and audit arrangements are closely associated with programme success. Clear and regular monitoring requirements are valuable, provided they are not unduly burdensome to participants and that the information collected serves a clear purpose in informing programme managers and is readily available for the purpose of evaluation. Information collected 'because it is there' places a double burden on those providing it and those responsible for its collection and collation. [Major finding in several programmes].
13Management and audit processes should place an equal emphasis on all the major programme objectives and ensure there is a balance between them through the project lifecycle.
14At all stages of the programme/project lifecycle, bureaucracy should be minimised. Thus, application processes should be efficient, notification of progress and payment should proceed without undue delays, reporting requirements should be effective (see above), etc. [Key finding in several programmes].
15Brand recognition: in some cases, the fostering of a strong and positive brand image was found to increase the attractiveness of the scheme to high-quality participants, to increase motivation through a sense of 'belonging' and recognition and to increase the likelihood of additional networking throughout the programme rather than between the collaborating partners alone. Labelling of graduate placements created a sense of community, with additional networking benefits and increased recognition and awareness with future employers. Various approaches are used for brand enhancement such as programme websites, workshops and newsletters. [E.g. UK KTP programme].
5.4 Lessons for evaluation and need for future research
16Evaluations must be sensitive to the complexity of the collaboration process, and the diversity of motivations, rationales, activities, outputs, outcomes and effects which it entails. A tendency to focus on quantifiable outputs should be minimised and attention focused on the behavioural change induced by the collaboration process and the intangible outcomes generated. The unanticipated outcomes of collaboration may also be more significant that those anticipated at the outset. It is also important to assess whether participants' motivations have been achieved although care must be taken with such approaches as respondents often tend to give the responses they consider to be expected by programme managers in the anticipation of further funding. [Common finding from several programmes].
17As ever, the issue of timing can be critical. The immediate outcomes of a collaboration can be quite different from the longer term effects produced through, for instance, capability building or skills enhancement. A propensity for further collaboration may lead to significant advances outside the support programme itself. It is clear that the management and procedural aspects of programme implementation imply quite different informational needs to those of output and impact assessment. Therefore, assuming that a long-term commitment to programme support has been made, a series of evaluations could be planned in turn examining management issues and outcome-oriented issues as the programme progresses. [Common finding from several programmes].
18The issue of causality is a particular methodological problem frequently encounters in the literature and evaluation reports (especially the CIS based studies and some of the survey based studies). Where possible, evaluations should be designed to establish not only the quantitative facts about impacts but also endeavour to explain the qualitative links between statistically linked variables. There is often a tendency to treat collaborative processes as an effective black box without due regard to the internal processes that generate particular results and outcomes. [See Section 4.5.4]
19As noted in point 12, the task of evaluation is made easier, and its associated costs reduced, if appropriate monitoring and data collation processes are in place through the lifetime of the programme, provided these are designed in such a way as not to add undue bureaucratic weight and barriers to participation. [Finding in several programme evaluations].
References
Abreu, M., Hughes, A., Grinevich, V., Kitson, M., 2009. Knowledge Exchange between Academics and the Business, Public and Third Sectors. CBR and UK Innovation Research Centre.
Afcha Chávez, S.M., 2011. Behavioural additionality in the context of regional innovation policy in Spain. Innovation: Management, Policy and Practice 13, 95-110.
Allen Consulting, 2005. The Economic Impact of Cooperative Research Centres in Australia: Delivering benefits for Australia. Cooperative Research Centres Association Inc.
Arranz, N., Fdez. de Arroyabe, J.C., 2008. The choice of partners in R&D cooperation: An empirical analysis of Spanish firms. Technovation 28, 88-100.
Autio, E., Kanninen, S., Gustafsson, R., 2008. First- and second-order additionality and learning outcomes in collaborative R&D programs. Research Policy 37, 59-76.
Bayona-Sáez, C., García-Marco, T., 2010. Assessing the effectiveness of the Eureka Program. Research Policy 39, 1375-1386.
Belderbos, R., Carree, M., Lokshin, B., 2004. Cooperative R&D and firm performance. Research Policy 33, 1477-1492.
Benfratello, L., Sembenelli, A., 2002. Research joint ventures and firm level performance. Research Policy 31, 493-507.
http://www.innovation-policy.org.uk/compendium/reference/Default.aspx?referenceid=568
Blumenthal, D., 2003. Academic-Industrial Relationships in the Life Sciences. New England Journal of Medicine 349, 2452-2459.
Blumenthal, D., Campbell, E.G., Causino, N., Louis, K.S., 1996. Participation of life science faculty in research relationships with industry. New England Journal of Medicine 335, 1734-1739.
Blumenthal, D., Causino, N., Campbell, E., Louis, K.S., 1996. Relationships between academic institutions and industry in the life sciences - An industry survey. New England Journal of Medicine 334, 368-373.
Blumenthal, D., Gluck, M., Seashore Louis, K., 1986. University-industry research relationships in biotechnology: Implications for the university. Science 232, 1361-1366.
Bozeman, B., Gaughan, M., 2007. Impacts of grants and contracts on academic researchers' interactions with industry. Research Policy 36, 694-707.
Branstetter, L., Sakakibara, M., 1998. Japanese research consortia: A microeconometric analysis of industrial policy. Journal of Industrial Economics 46, 207-233.
Branstetter, L.G., Sakakibara, M., 2002. When do research consortia work well and why? Evidence from Japanese panel data. American Economic Review 92, 143-159.
Bruno, N., Giarracca, F., Horvath, A., Kamburow, T., Navarrete Moreno, C., Parker-Rhodes, L., Simmonds, P., 2011. Evaluation of innovation activities: methods and practices - Report of the Literature Review. European Commision DG Regional Policy, Brussels.
Buisseret, T.J., Cameron, H.M., Georghiou, L., 1995. What Difference Does It Make - Additionality in The Public Support Of R&D In Large Firms. International Journal of Technology Management 10, 587-600.
Busom, I., Fernández-Ribas, A., 2008. The impact of firm participation in R&D programmes on R&D partnerships. Research Policy 37, 240-257.
CBR, PACEC, 2009a. Evaluation of the Effectiveness and Role of HEFCE/OSI Third Stream Funding: A report to HEFCE. PACEC and the Centre for Business Research, University of Cambridge.
CBR, PACEC, 2009b. The Evolution of the Infrastructure of the Knowledge Exchange System: A report to HEFCE. PACEC and the Centre for Business Research, University of Cambridge.
CBR, PACEC, 2010a. The Higher Education Knowledge Exchange System in the United States. A report to HEFCE. PACEC and the Centre for Business Research, University of Cambridge.
CBR, PACEC, 2010b. The Intellectual Property Regime and its Implications for Knowledge Exchange: A report to HEFCE. PACEC and the Centre for Business Research, University of Cambridge.
CBR, PACEC, 2010c. Knowledge Exchange and the Generation of Civic and Community Impacts: A report to HEFCE. PACEC and the Centre for Business Research, University of Cambridge.
CBR, PACEC, 2010d. Synergies and Trade-offs Between Research, Teaching and Knowledge Exchange: A report to HEFCE. PACEC and the Centre for Business Research, University of Cambridge.
Clarysse, B., Wright, M., Mustar, P., 2009. Behavioural additionality of R&D subsidies: A learning perspective. Research Policy 38, 1517-1533.
Cosh, A., Fu, X., Hughes, A., 2005. Management characteristics, collaboration and innovative efficiency: evidence from UK survey. ESRC Centre for Business Research, University of Cambridge.
Cosh, A.D., Hughes, A., 2007. British Enterprise: Thriving or Surviving? Centre for Business Research, University of Cambridge.
Czarnitzki, D., Ebersberger, B., Fier, A., 2007. The relationship between R&D collaboration, subsidies and R&D performance: Empirical evidence from Finland and Germany. Journal of Applied Econometrics 22, 1347-1366.
Danish Agency for Science Technology and Innovation, 2011. Economic Impact of International Research and Innovation Cooperation: Analysis of 25 years of Danish participation in EUREKA.
Davenport, S., Grimes, C., Davies, J., 1998. Research collaboration and behavioural additionality: A New Zealand case study. Technology Analysis and Strategic Management 10, 55-67.
DTI, 1995. Evaluation of EUREKA industrial and economic effects in the UK. DTI, London.
Dyer, J.H., Powell, B.C., 2001. Determinants of Success in ATP-Funded R&D Joint Ventures A Preliminary Analysis Based on 18 Automobile Manufacturing Projects. NIST GCR 00-803.
E.G.P Applied Economics, 2011. Estimating the Effect of Participation in Eureka on Firm Performance. EUREKA.
Etzkowitz, H., 1998. The norms of entrepreneurial science: cognitive effects of the new university-industry linkages. Research Policy 27, 823-833.
EUREKA, 1985. Hannover Declaration: Declaration of Principles Relating to EUREKA, Hannover.
EUREKA, 2004. SMEs IN INTERNATIONAL COOPERATION: Key Factors for Success. EUREKA.
EUREKA, 2005a. 20 Years of Pan-European Innovation. Eurekanews.
EUREKA, 2005b. The Impact of EUREKA: Two decades of support for European innovation.
EUREKA, 2006. Eureka - A Cornerstone of the European Research Innovation Area.
EUREKA, 2011. EUREKA Strategic Roadmap 2010-2014: Towards a strong role of EUREKA in the European Research Area (ERA).
EUREKA, 2012. 25 year country overviews: 1985 – 2011. EUREKA, Brussels.
European Commission, 2003. Raising EU R&D Intensity: Improving the Effectiveness of Public Support Mechanisms for Private Sector Research and Development DIRECT MEASURES. European Commission EUR 20716, Belgium.
Fernandez-Ribas, A., Shapira, P., 2009. The role of national and regional innovation programmes in stimulating international cooperation in innovation. International Journal of Technology Management 48, 473-498.
Frenken, K., Ponds, R., van Oort, F., 2010. The citation impact of research collaboration in science-based industries: A spatial-institutional analysis. Papers in Regional Science 89, 351-271.
Frenz, M., Ietto-Gillies, G., 2009. The impact on innovation performance of different sources of knowledge: Evidence from the UK Community Innovation Survey. Research Policy 38, 1125-1135.
Georghiou, L., 1998. Issues in the Evaluation of Innovation and Technology Policy, in: OECD (Ed.), Policy Evaluation in Innovation and Technology: Towards Best Practices. OECD Publishing, Paris.
Georghiou, L., 1999. Socio-economic Effects of Collaborative R&D—European Experiences. The Journal of Technology Transfer 24, 69-79.
Georghiou, L., 2007. What lies beneath: Avoiding the risk of under-evaluation. Science and Public Policy 34, 743-752.
Georghiou, L., Clarysse, B., 2006. Introduction and Synthesis, in: OECD (Ed.), Government R&D Funding and Company Behaviour: Measuring Behavioural Additionality. OECD Publishing, Paris.
Godin, B., Gingras, Y., 2000. Impact of collaborative research on academic science. Science and Public Policy 27, 65-73.
Gök, A., 2010a. An Evolutionary Approach to Innovation Policy Evaluation: Behavioural Additionality and Organisational Routines. PhD Thesis. Manchester Institute of Innovation Research, The University of Manchester, Manchester.
Gök, A., 2010b. Organisational Routines and Behavioural Additionality: Abehavioural Approach to Innovation Policy Evaluation. PhD. Manchester Institute of Innovation Research, University of Manchester, Manchester.
Gök, A., Edler, J., 2010. Exploring the Use of Behavioural Additionality, in: Edler, J. (Ed.), INNO-Appraisal: Understanding Evaluation of Innovation Policy in Europe. Report to EC DG Enterprise and Industry, Brussels.
Guy, K., Georghiou, L., Quintas, P., Cameron, H., Hobday, M., Ray, T., 1991. Evaluation of the Alvey Programme for Advanced Information Technology. HMSO.
Hagedoorn, J., Link, A.N., Vonortas, N.S., 2000. Research partnerships. Research Policy 29, 567-586.
Howard Partners, 2003. Evaluation of the Cooperative Research Centres Programme. Australia Department of Education, Science and Training.
Howells, J., Nedeva, M., Georghiou, L., Evans, J., Hinder, S., 1998. Industry-Academic Links in the UK. HEFCE Ref 98/70, Manchester.
Hughes, A., 2008. Entrepreneurship and Innovation Policy: Retrospect and Prospect. The Political Quarterly 79, 133-152.
Insight Economics, 2006. Economic Impact Study of the CRC Programme. Australian Government Department of Education, Science and Training.
Kang, K.N., Park, H., 2012. Influence of government R&D support and inter-firm collaborations on innovation in Korean biotechnology SMEs. Technovation 32, 68-78.
Katz, J.S., Hicks, D., 1997. How much is a collaboration worth? A calibrated bibliometric model. Scientometrics 40, 541-554.
Kleinknecht, A., Reijnen, J.O.N., 1992. Why do firms cooperate on R&D? an empirical study. Research Policy 21, 347-360.
Lambert, R., 2003. Lambert Review of Business-University Collaboration. HM Treasury, London.
Lebeau, L.M., Laframboise, M.C., Larivière, V., Gingras, Y., 2008. The effect of university-industry collaboration on the scientific impact of publications: The Canadian case, 1980-2005. Research Evaluation 17, 227-232.
Lee, S., Bozeman, B., 2005. The impact of research collaboration on scientific productivity. Social Studies of Science 35, 673-702.
Lemola, T., Lievonen, J., 2008. The role of innovation policy in fostering open innovation activities among companies.
Martin and Associates, Vision Consulting, Nauwelaers, C., 2004. Innovation Networks. Forfas.
Mercer, D., Stocker, J., 1998. Review of Greater Commercialisation and Self Funding in the CRC Programme. AUSTRALIAN DEPARTMENT OF INDUSTRY, SCIENCE AND TOURISM.
Miotti, L., Sachwald, F., 2003. Co-operative R&D: Why and with whom? An integrated framework of analysis. Research Policy 32, 1481-1499.
Mohnen, P., Hoareau, C., 2003. What type of enterprise forges close links with universities and government labs? Evidence from CIS 2. Managerial and Decision Economics 24, 133-145.
Mortimer, D., 1997. Going for Growth Business Programs for Investment, Innovation and Export. Commonwealth of Australia, Canberra.
Mothe, C., Quelin, B.V., 1999. Creating new resources through European R and D partnerships. Technology Analysis and Strategic Management 11, 31-43.
Mothe, C., Quelin, B.V., 2000. Creating Competencies Through Collaboration: The Case of EUREKA R & D Consortia. European Management Journal 18, 590-604.
Myers, R., 1995. Changing Research Culture, Australia - 1995: Report of the CRC Programme Evaluation Steering Committee,, Jul 1995. Australia Government Publishing Service.
O'Doherty, D., 1998. Networking in Ireland—Policy responses, NESC Research Series: Sustaining Competitive Advantage, Proceedings of National Economic and Social Council (NESC) Seminar, Dublin.
O'Kane, M., 2008. Collaborating to A Purpose: Review of the Cooperative Research Centres Program. Commonwealth of Australia.
OECD, 2001. Innovative Networks: Co-operation in National Innovation Systems. OECD Publishing, Paris.
OECD, 2004. Public-Private Partnership for Research and Innovation: an Evaluation of the Australian Experience. OECD: Paris.
OECD (ed.), 2006. Government R&D Funding and Company Behaviour: Measuring Behavioural Additionality. OECD Publishing, Paris.
OECD (ed.), 2007. Science, Technology And Innovation Indicators In A Changing World: Responding to Policy Needs. OECD Publishing.
OECD, EUROSTAT, 2005. OECD Proposed Guidelines for Collecting and Interpreting Technological Innovation Data: Oslo Manual 3rd Edition. OECD, Paris.
PACEC, 2011. Evaluation of the Collaborative Research and Development Programmes: Final Report.
Peterson, J., 1993. Assessing the performance of European collaborative R&D policy: The case of Eureka. Research Policy 22, 243-264.
Productivity Commission, 2007. Public Support for Science and Innovation. Productivity Commission of Australia.
Regeneris Consulting Ltd., 2010. Knowledge Transfer Partnerships Strategic Review. Technology Strategy Board.
Roessner, D., 2000. Outcomes and Impacts of the State/Industry-university Cooperative Research Centres (S/IUCRC) Programme: Final Report. SRI International.
Roessner, J.D., Cheney, D.W. and Coward, H.R., 2004. Impact on Industry of Interactions with Engineering Research Centres – Repeat Study: Summary Report. SRI International.
Rothaermel, F.T., Agung, S.D., Jiang, L., 2007. University entrepreneurship: a taxonomy of the literature. Industrial and Corporate Change 16, 691-791.
Ruegg, R., Feller, I., 2003. A Toolkit for Evaluating Public R&D Investment Models, Methods, and Findings from ATP's First Decade. National Institute of Standards and Technology, Gaithersburg.
Sáez, C.B., García Marco, T., Arribas, E.H., 2002. Collaboration in R & D with universities and research centres: An empirical study of Spanish firms. R and D Management 32, 321-341.
Sakakibara, M., 1997. Evaluating government-sponsored R & D consortia in Japan: Who benefits and how? Research Policy 26, 447-473.
Sakakibara, M., 2001. Cooperative research and development: Who participates and in which industries do projects take place? Research Policy 30, 993-1018.
Sakakibara, M., Cho, D.S., 2002. Cooperative R&D in Japan and Korea: A comparison of industrial policy. Research Policy 31, 673-692.
Segarra-Blasco, A., Arauzo-Carod, J.M., 2008. Sources of innovation and industry-university interaction: Evidence from Spanish firms. Research Policy 37, 1283-1295.
Sevon, G., Kreiner, K. (ed.), 1998. Constructing R and D Collaboration: Lessons from European EUREKA Projects. Copenhagen Business School, Copenhagen
Smith, D., Griffin, M., O'Neil, P., Rees, J., Skingle, M., Stewart, W., Yarrow, D., 2003. Strategic Review of LINK Collaborative Research. Report of the Independent Review Panel.
Smith, K., 2004. Measuring Innovation, in: Fagerberg, J., Mowery, D.C., Nelson, R.R. (Eds.), The Oxford Handbook of Innovation. Oxford University Press, Oxford.
SQW Ltd., 2002. Evaluation of the Teaching Company Scheme, Report to Small Business Service. DTI Evaluation Report Series, No. 7.
Teirlinck, P., Spithoven, A., 2010. Fostering industry-science cooperation through public funding: differences between universities and public research centres. Journal of Technology Transfer, 1-20.
Tether, B.S., 2002. Who co-operates for innovation, and why An empirical analysis. Research Policy 31, 947-967.
Tsipouri, L., Reid, A., Arundel, A., Hollanders, H., 2006. European Innovation Progress Report. European Commission DG Enterprise and Industry, Brussels.
Tsipouri, L., Reid, A., Mannik, K., Rannala, R., Ruiz Yaniz, M., Bruno, N., Elci, S., 2009. European Innovation Progress Report. European Commission DG Enterprise and Industry, Brussels.
Watanabe, C., Kishioka, M., Nagamatsu, A., 2004. Effect and limit of the government role in spurring technology spillover - A case of R&D consortia by the Japanese government. Technovation 24, 403-420.
Wilson, T., 2012. A Review of Business-University Collaboration. BIS, London.
Annex 1: Collaboration schemes from which evidence was drawn
| Programme |
Agency/Country |
Brief overview |
| Alvey Programme |
(former) Department of Trade and Industry, Ministry of Defence, (former) Science and Engineering Research Council, UK |
Collaborative R&D Programme in IT. Aimed to support the academic and industrial science and technology base, transfer academic know-how to industry, and enhance the competitive potential of the sector, focusing on pre-competitive R&D with some elements nearer to market. |
| Advanced Technology Program (ATP) |
National Institute of Standards and Research, USA |
Designed to stimulate early stage advanced technology development that would otherwise not be fundable, aimed at early stage research in industry - supported academia indirectly (as subcontractors or collaborators in projects). |
| Cooperative Research and Development Agreements (CRADAS) |
US Federal Government |
Partnerships in which a private sector partner and a national laboratory work together to further R&D. |
| State/Industry University Cooperative Research Centres Programme |
National Science Foundation, USA |
Intended to promote industry-university cooperation in research, and to incorporate the interests of state governments. |
| Engineering Research Centres Programme |
National Science Foundation, USA |
A government-industry-university partnership to strengthen the competitive position of US firms in world trade |
| Collaborative Research and Development Programmes |
(now) Technology Strategy Board, UK |
Designed to assist the industrial and research communities to work together on R&D projects in strategically important areas of science, engineering and technology. |
| LINK |
UK Research Councils and Government Departments |
Long-standing collaborative support scheme for the promotion of collaboration in pre-commercial research between businesses and the research base. |
| Knowledge Transfer Partnerships |
Technology Strategy Board, UK |
Well-established collaboration programme enabling companies to obtain knowledge, technology or skills which they consider to be of strategic competitive importance, from the further/higher education sector or from a research and technology organisation. Projects are undertaken by good quality individuals recruited for the purpose who works within the company. |
| VINN Excellence Centres |
VINNOVA, Sweden |
To build bridges between science and industry by creating excellent academic research environments in which industrial companies participate actively and persistently in order to derive long-term benefits. |
| Berzelii Centres |
VINNOVA, Sweden |
Focus on excellent basic research, and to collaborate actively with stakeholders from the private and public centres and to put research results to concrete use in the form of commercial applications. |
| Centres for Research-Based Innovation (SFI) |
Research Council of Norway |
Intended to promote innovation by supporting long-term industrially oriented research and forging close alliances between research-active enterprises and prominent research groups. The scheme is also expected to enhance technology transfer, internationalization and researcher training. |
| Danish Innovation Consortium Scheme |
Danish Agency for Science, Technology and Innovation |
An innovation consortium (IC) is a flexible framework for collaboration between companies, research institutions and non-profit advisory/knowledge dissemination parties. An IC must consist of at least two companies, one research institution and one advisory and knowledge dissemination party. Additionally, it may involve or attach other types of partners that are considered relevant to the project. |
| Programme for the Development of Industrial |
General Secretariat of Research and Technology, Greece |
Scheme for industrial research launched in the cohesion context. One achievement was the strengthening of research collaboration among firms and between firms and university/research centres. |
| Cooperative Research Centres Programme |
Commonwealth Government, Australia |
To enhance industrial, commercial and economic growth through the development of sustained, user-driven, cooperative public-private research centres that achieve high levels of outcomes in adoption and commercialisation. Also to address research concentration for world-class teams and preparing PhD graduates for non-academic careers. |
| Innovation Partnerships Programme |
Forfas, Ireland |
Aim to build R&D partnerships between industry and the Third Level Institutions. |
| Advanced Manufacturing Technology Programme |
Forfas, Ireland |
Provides technology transfer and consultancy services in Advanced Manufacturing Technology to both indigenous firms and to multinationals located in Ireland |
| Technology for Business Growth |
Foundation for Research, Science and Technology, New Zealand |
Supports collaborative R&D in order to develop technologies that will improve business performance and profit, and to diffuse them to New Zealand's companies to improve their technological literacy and abilities. The programme, inter alia, attempts to improve the linkages between the private sector and government research institutes. |
| Concerted Projects under the National R&D Plan |
Ministry of Science and Innovation, Spain |
To promote R&D activities in firms and to encourage cooperation between the latter and public research centres, including universities. |
| R&D alliances |
Industrial Technology Research Institute, Ministry of Economic Affairs, Taiwan |
Bring together firms, and public sector research institutes, plus trade associations, and catalytic financial assistance from government. |
| National Technology Programmes |
Tekes, Finland |
Collaborative programmes targeted at industrial technological development in selected technology sectors, through technology development and diffusion. |
